Sampling: noise to picture

Walking downhill in probability, one noisy step at a time

The forward process defines a chain of distributions $q(x_t)$ that starts at the data and ends at noise. The reverse process tries to walk that chain backwards. At each step the network estimates the noise $\varepsilon_\theta(x_t,t)$, and that estimate converts into an estimate of the clean point, $\hat{x}_0 = (x_t - \sqrt{1-\bar\alpha_t}\,\varepsilon_\theta)/\sqrt{\bar\alpha_t}$. A small step is then taken toward that estimate, scaled down so it does not overshoot the noise that is still present.

Every sampler in this volume is a variation on that one move, and they differ in one question: how much randomness is re-injected at each step? Re-inject the full amount dictated by the forward process and you get the ancestral sampler; re-inject none and you get a deterministic ordinary differential equation; re-inject something in between and you get a whole family in between.

The canvas below shows the target distribution as a blue density — three Gaussian modes — with the sampler's trajectory drawn over it in pink. The black dot is the starting noise. Press Reseed for a different start, switch methods, and drag the step slider to watch the path get coarser or finer.

Add the noise back, on purpose

The ancestral sampler is the reverse process written down literally. At each step it computes the mean of $q(x_{t-1}\mid x_t)$ using the network's noise estimate, and then samples around that mean with the variance the forward process prescribes. In symbols, with $\beta_t$ the step's noise and $\bar\alpha$ the cumulative signal,

$$ x_{t-1} = \frac{1}{\sqrt{\alpha_t}}\Big(x_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\,\varepsilon_\theta(x_t,t)\Big) + \sigma_t z, \qquad z\sim\mathcal{N}(0,I) $$

That final $\sigma_t z$ is the part people find strange. The model has just estimated the noise, so why put noise back? Because the goal is not to recover one particular clean point — that information is gone — but to sample from the conditional distribution. The true reverse distribution is not a spike at its mean; it has a spread, and ignoring that spread produces samples that are too smooth and too similar to one another. Injecting the prescribed variance is what keeps the sampler honest about its own uncertainty.

Switch the demo to DDPM and press Reseed a few times. The same starting noise gives a different trajectory and a different endpoint every time, and the faint runs on the canvas show why: the randomness at each step is a real degree of freedom. This is also the property that makes DDPM expensive in practice, because the noise must be integrated carefully across the whole trajectory for the sample to be valid.

Set the variance to zero and lose nothing but the dice

DDIM is what happens when you notice that the reverse process does not have to be Markovian or stochastic. The same network, evaluated at a sparse set of timesteps, can define a deterministic map from noise to sample:

$$ x_{t-1} = \sqrt{\bar\alpha_{t-1}}\,\hat{x}_0 + \sqrt{1-\bar\alpha_{t-1}}\,\varepsilon_\theta(x_t,t) $$

Read it as two endpoints: reconstruct the clean point $\hat{x}_0$, then re-noise it to the next noise level using the network's own noise estimate. There is no $\sigma_t z$, so the whole trajectory is fixed once the starting noise is fixed. Run the sampler twice with the same seed and you get bit-identical output, which is why DDIM is the usual choice for interpolation, inversion and editing: the map is a function, and functions can be differentiated and inverted.

The histogram below is the experiment that makes the difference unmistakable. It takes one fixed starting noise and runs the reverse process 120 times, recording the final $x$-coordinate each time. Under DDIM every run follows the same path, so all 120 land on the same value and the histogram is a single spike. Under DDPM each run draws fresh noise at every step, so the endpoints scatter around the mode — the same fixed start has become a distribution.

A useful way to hold the two samplers in your head: DDPM integrates a stochastic differential equation with the noise left in, and DDIM integrates the probability-flow ODE that has the same marginals. They agree in the limit of infinitely many steps and diverge in how they spend a finite budget, and both are special cases of the $\eta$-parameterised family where $\eta$ interpolates between them.

The slider everyone wants to move left

Training touches a thousand noise levels; sampling can use far fewer because DDIM does not need to follow the chain one step at a time. Drag the slider down to ten steps and the trajectory becomes a jagged polyline that hops between noise levels; drag it to two hundred and it hugs a smooth curve. The sample quality follows the same pattern: too few steps and the path cuts corners, missing modes or overshooting them, while past a point extra steps cost compute and change almost nothing.

This is the central engineering trade of diffusion inference. Every step is one forward pass of the network, so halving the steps halves the latency, and the entire distillation literature exists to buy back the quality lost. The number that matters is not the count of steps but the size of the jump each step takes in noise level, which is why schedules with rapidly changing SNR need denser sampling there.

There is a subtlety the demo makes visible. Because DDIM is deterministic, a coarse run does not average out its own error over many stochastic steps — it commits to a path, and if that path is wrong it stays wrong. Ancestral DDPM has more chances to correct itself but pays in noise. Practical systems often use a deterministic sampler with a good schedule and accept a modest step count, because the wall-clock is what users feel.

Sampler choice is a product decision

Two threads lead out of this part. The first is the score function, which explains why the network's noise estimate is also a gradient and why the deterministic and stochastic samplers describe the same object; the second is the machinery of conditioning, which turns an unconditional sampler into one that follows a prompt. The next part takes the first thread.

Further reading

The sampling literature is where diffusion stopped being slow. These four papers are the arc: the ancestral sampler, the deterministic one, the unification, and the distillation that made interactive generation possible.

If you read one, read DDIM, because it is the paper that reframed sampling as an ODE and every fast sampler since has been a variation on that idea.

• Jonathan Ho, Ajay Jain and Pieter Abbeel, "Denoising Diffusion Probabilistic Models", 2020 — the ancestral reverse process.
• Jiaming Song, Chenlin Meng and Stefano Ermon, "Denoising Diffusion Implicit Models", 2020 — the deterministic sampler and the $\eta$ family.
• Yang Song, Jascha Sohl-Dickstein and coauthors, "Score-Based Generative Modeling through Stochastic Differential Equations", 2020 — the probability-flow ODE that DDIM discretises.
• Tim Salimans and Jonathan Ho, "Progressive Distillation for Fast Sampling of Diffusion Models", 2022 — halving the step count by distillation.
• Alexia Jolicoeur-Martineau and coauthors, "Gotta Go Fast When Generating Data with Score-Based Models", 2021 — practical step-count selection and adaptive solvers.

Cheat sheet