Conditioning and classifier-free guidance

Getting words into a noise predictor

The prompt is turned into vectors by a text encoder: CLIP for its familiar embedding, T5 for longer and more compositional prompts, or a vision-language model when the model needs to reason about the caption. The encoder output is a sequence of token features, one per token, and the question is how those features reach the denoiser's spatial features.

There are two answers, and the difference matters. In cross-attention, the image features supply the queries and the text features supply keys and values, so each patch of the latent attends to the tokens it finds relevant. In joint attention, image and text tokens are concatenated into one sequence and attend to each other with shared weights; MMDiT's separate streams are a refinement of this, keeping per-modality projections but sharing the attention. Cross-attention is cheaper and was the original design; joint attention conditions more strongly and scales better with long prompts.

Train one model to do two jobs

Classifier guidance once required a separate noisy classifier whose gradient nudged samples toward a class. Classifier-free guidance gets the same effect without the classifier by training a single model to predict both a conditional and an unconditional noise, dropping the conditioning randomly during training so the model learns the unconditional task as well. At sampling time you run the model twice on the same input — once with the prompt and once without — and combine the two predictions linearly:

$$ \varepsilon \;=\; w\,\varepsilon_\text{cond} \;+\; (1-w)\,\varepsilon_\text{uncond}. $$

When $w = 1$ this is pure conditional prediction. When $w = 0$ it is the unconditional model, a plain sample from the data distribution. Values between and beyond those two points are the interesting part: the combination is a linear extrapolation, not an interpolation, because the coefficient on the unconditional term goes negative once $w > 1$. Subtracting a multiple of the unconditional direction pushes the sample away from the generic and toward whatever makes the prompt distinctive.

In code this is Diffusion.cfg(w, uncond, cond), applied componentwise to the noise prediction at every step. It costs one extra forward pass per step, which is the entire price of guidance.

Diversity collapses, then oversaturation sets in

The demo below is a two-dimensional stand-in for the whole mechanism. The data distribution is a mixture of three modes; the conditional target is a single one of them; the unconditional target is the mixture. (The unconditional predictor here is the mixture's posterior mean blended slightly toward the global mean — the small approximation error a real unconditional model carries, and the reason guidance has anything to extrapolate at all. With two perfect predictors the guided field would vanish at the mode and no scale could push past it.) Every sample starts somewhere in the mixture and is then moved by the guided prediction, so you can see in one picture what the scalar does. At $w = 0$ the samples stay spread across the mixture; near $w = 1$ they collapse onto the conditional mode; past that the extrapolated fixed point moves through and beyond the mode, and the cloud visibly overshoots it.

The readout is Diffusion.guidanceEffect, which returns the guided value alongside how far past the conditional prediction the result has been extrapolated. Watch that excess grow linearly in $w$ while the sample cloud first tightens and then scatters: that is the same oversaturation an image model shows as washed-out, over-contrasted frames at scale 15.

Two fixes for the artifacts guidance introduces

High guidance makes the predicted noise too large, which is the mechanism behind over-saturation. Guidance rescaling measures the standard deviation of the guided prediction and scales it back toward the conditional prediction's, so the direction is preserved while the magnitude is restrained. It is the line Diffusion.cfgRescale(x, ratio), and it is why some pipelines expose a rescale parameter next to the guidance scale.

The second fix is to apply guidance only where it helps. Several models find that full guidance late in the trajectory harms fine detail, while early and mid-trajectory guidance is what fixes the composition. A guidance interval leaves the scale at a baseline outside a chosen band of timesteps and switches to the guided value inside it — the step function Diffusion.cfgInterval(wBase, w, t, tLo, tHi) plots the weight actually used.

One scalar, every pipeline

The knob is universal because the trick is universal: run the model with and without conditioning, then extrapolate. Any conditional generative model with an unconditional fallback can be guided this way, and almost every one that ships is.

Further reading

Ho and Salimans introduced classifier-free guidance as a way to remove the auxiliary classifier; Dhariwal and Nichol then showed it beating classifier guidance outright. The rescale trick and the interval schedule come from later production experience, and the MMDiT paper is the clearest account of joint versus cross attention at scale.

Read Ho and Salimans first for the formula, Dhariwal and Nichol for the empirical case, Lin and coauthors for rescaling, and Esser and coauthors for the attention variant.

• Jonathan Ho and Tim Salimans, "Classifier-Free Diffusion Guidance", 2022 — the formula, and training one model for both conditional and unconditional prediction.
• Prafulla Dhariwal and Alex Nichol, "Diffusion Models Beat GANs on Image Synthesis", 2021 — classifier guidance and the case against it.
• Shanchuan Lin, Bingchen Liu, Jiashi Li and Xiao Yang, "Common Diffusion Noise Schedules and Sample Steps are Flawed", 2023 — variance-preservation fixes, including guidance rescaling.
• Patrick Esser and coauthors, "Scaling Rectified Flow Transformers for High-Resolution Image Synthesis", 2024 — joint attention and the MMDiT conditioning design.

Cheat sheet