Skip to content
Phobos

Random numbers

Random numbers without a generator’s state: each value is a hash of a seed and the element’s own counter, so a tile of them is a few integer instructions an element, fused into the statement that uses them. The same seed and counter give the same number, which is how a backward pass redraws the dropout mask of its forward pass instead of storing it.

@autotune(R in [16])
kernel dropout_rows(X: tensor<f32>[M, 256], Y: tensor<f32>[M, 256], seed: i32) {
  let r = ctx.block.x * R
  Y[r :+ R, :] = dropout(X[r :+ R, :], 0.1, seed, (r + iota(0)) * 256 + iota(1))
}

The counter is each element’s flat index in the whole tensor, not in the block, so no two elements of a launch draw the same number. A fresh draw takes a fresh seed, or counters past the ones used: randn reads the uniforms at 2 c and 2 c + 1, so a kernel that takes both uniform and randn from one seed gives them different seeds. A kernel that passes one counter to several calls can hold it in a tile, var c: tile<i32>[R, 256] = (r + iota(0)) * 256 + iota(1), at the cost of the tile’s shared memory; a let cannot name it, since iota is a position only in the value of an assignment.

Counter-based random numbers on a tileA tile of eight rows of 32 elements, twelve rows of 16 on a narrow screen, each shaded by the value the standard library's uniform, randn or dropout gives its counter, the element's flat index, under seed 1, 2 or 3. Uniform shades by the value in [0, 1), randn by the value from -3 to 3, and dropout fills the kept elements and leaves the dropped ones empty. The values are computed from the library's Phobos source.FIG / STDLIB / RANDOMYc = (r + iota(0)) * 32 + iota(1)uniform(seed, c)

uniform: 256 counters, mean 0.513; each a multiple of 2^-23 in [0, 1).

  • per element, in registers
Computed from the definitions in phobos-lang/src/stdlib/random.ph, with the device's wrapping i32 arithmetic. The same seed and counters draw the same tile on every run.

uniform

Section titled “uniform”
fn uniform(seed, counter)expression

A number in [0, 1) for each counter, uniform on a grid of 2^-23: the hash's low 23 bits.

Source, std/random.ph:21
fn uniform(seed, counter) = f32(hash32(counter ^ hash32(seed)) & 8388607) * 1.1920928955078125e-7

randn

Section titled “randn”
fn randn(seed, counter)expression

A standard normal number for each counter, the Box-Muller transform of the uniform pair at counters 2 counter and 2 counter + 1. The uniform's grid ends its tails near 5.6.

Source, std/random.ph:26
fn randn(seed, counter) = sqrt(-2.0 * log(1.0 - uniform(seed, 2 * counter))) * cos(6.283185307179586 * uniform(seed, 2 * counter + 1))

dropout

Section titled “dropout”
fn dropout(x, p, seed, counter)expression

Dropout: each element of x zeroed with probability p and the kept ones scaled by 1 / (1 - p), so the expectation is x. p in [0, 1).

Source, std/random.ph:30
fn dropout(x, p, seed, counter) = where(uniform(seed, counter) >= p, x / (1.0 - p), 0.0)

hash32

Section titled “hash32”
fn hash32(x)expression

The lowbias32 hash of each element of the i32 value x.

Source, std/random.ph:13
fn hash32(x) = _mix(_mix(_mix(x, 16) * 2146121005, 15) * -2073254261, 16)

The hash runs in wrapping i32 arithmetic, which a counter built from iota and integers computes in, as iota does. Its shifts are logical: >> is arithmetic on a signed integer, so the bits it copies in are masked off.