What audio is, numerically

One axis, sampled fast

A waveform is a function of time: at each instant the microphone reports the air pressure, positive or negative relative to the ambient level. To store it on a computer you sample that function at regular intervals and keep the values. The number of samples per second is the sample rate, written $f_s$, and the time between samples is $1/f_s$. Speech telephony uses eight kilohertz, wideband speech sixteen, music and high-fidelity speech forty-four point one or forty-eight, and the models in this chapter usually work at twenty-four. The demo below draws a sum of three tones — a fundamental and two partials, which is what makes it read as a chord rather than a beep — and lets you change its pitch and its length.

Nothing about the picture is digital yet: the list is a faithful-enough approximation to a continuous wave, provided we sampled fast enough, which is the subject of the next step. What matters here is only that the signal is an array of scalar values with a single index. A colormap of it would be a one-pixel-wide image; a spectrogram of it will be a two-dimensional picture, and that is the representation every audio model actually consumes.

The one theorem you cannot skip

Sampling at $f_s$ can only represent frequencies up to the Nyquist frequency $f_s/2$. A component above that limit is not lost cleanly; it folds back and appears as a lower frequency, an artefact called aliasing, the same way a wagon wheel on film can appear to spin backwards. This is why a real pipeline low-pass filters before it downsamples. It is also why the sample rate is a hard ceiling on what a model can ever represent: at 24 kHz nothing above 12 kHz exists in the data, and most of the perceptual detail above that ceiling is inaudible to adults anyway.

The spectrum is the other view of the same signal. A discrete Fourier transform of one window of samples reports, for each frequency bin, how much of that frequency is present. The canvas below takes a single short window of the chord and plots its magnitude against frequency, with a dashed marker at the fundamental. The peaks are the partials; the rest is leakage and the tail of the window. Almost every idea in audio generation is a statement about this picture rather than about the waveform, because the spectrum is where the structure lives.

A trade you cannot avoid

A single Fourier transform of the whole signal tells you which frequencies are present but not when. The short-time Fourier transform fixes that by transforming overlapping short windows and stacking the results: each window gives one column of a two-dimensional picture whose axes are time and frequency. The window is not rectangular — a smooth taper such as a Hann window reduces the spectral leakage that a hard cut would create — and the amount of overlap is the hop.

Window length sets a trade with no free side. A long window resolves frequency finely but smears time, so a plosive or a note onset is blurred; a short window pins down the moment but blurs neighbouring frequencies. The hop sets the frame rate independently: hop equals window over two is the usual starting point, and smaller hops oversample time at the cost of more frames. The readout tracks both, and you can drag the sliders to watch the number of frames and the frequency resolution move in opposite directions.

Frequency resolution is $f_s/\text{win}$ hertz per bin; time resolution is $\text{hop}/f_s$ milliseconds per frame. Their product is fixed by the sampling, which is the uncertainty principle wearing engineering clothes.

Compress frequency the way an ear does

Hertz are linear, but hearing is not. We resolve a few hertz near the bottom of the range and hundreds of hertz at the top, so a linear frequency axis spends most of its bins on detail nobody can hear. The mel scale rescales frequency so that equal distances sound equal: it is roughly linear below a kilohertz and logarithmic above. A mel filterbank is a bank of overlapping triangles spaced evenly on that scale, and applying it collapses hundreds of FFT bins into a few dozen bands. The heatmap below is the result — a mel spectrogram, the standard input to a speech recogniser and a common intermediate for a synthesiser.

What it throws away is resolution in the high frequencies and all phase. For recognition that is fine, indeed helpful, because it matches the ear and shrinks the input. For generation it is a loss you have to undo: a vocoder or a diffusion decoder has to invent the fine structure and the phase back from the compressed representation, which is exactly why the neural codec in the next part keeps a learned latent rather than a plain mel spectrogram.

Every audio model starts from these four pictures

The same machinery reappears outside speech. Music generation, audio editing and environmental-sound models all consume a spectrogram or a learned codec latent, and all of them are constrained by the sample rate and the STFT trade you can see above. The next part takes the one step this one left out: how a network learns its own front end instead of using a fixed filterbank.

Further reading

The sources below cover the classical signal processing first and the learned front ends second. If you take away one idea, take away the token-rate arithmetic: it explains why audio generation is a codec problem before it is a language-model problem.

Smith's Mathematics of the Discrete Fourier Transform is the free, careful treatment of the transform; Oppenheim and Schafer is the standard reference for the STFT and windowing; and the mel-scale original is worth reading once for the motivation rather than the constants. The EnCodec paper is the bridge into the next part.

• Julius O. Smith III, Mathematics of the Discrete Fourier Transform, online book — the DFT, the window and the leakage it controls.
• Alan Oppenheim and Ronald Schafer, Discrete-Time Signal Processing — sampling, aliasing and the short-time Fourier transform.
• Stevens, Volkmann and Newman, "A Scale for the Psychological Magnitude of Pitch", 1937 — the mel scale on which the filterbank is built.
• Défossez, Copet, Synnaeve and Adi, "High Fidelity Neural Audio Compression", 2022 — EnCodec, the codec of the next part.

Cheat sheet