Low-rank approximation and PCA
Most matrices that describe the real world are almost low-rank: a handful of directions carry nearly all of the signal, and the rest is noise or redundancy. This part makes that precise. Truncating the SVD keeps the k largest singular values and throws away the tail, and it does so optimally — no other rank-k matrix is closer to the original. Apply the very same truncation to a matrix of data points and you have principal component analysis: the directions of greatest variance are exactly the top right singular vectors.
The big idea: throw away the small singular values
A sum of rank-one layers, ranked by importance
Part 16 ended with the singular value decomposition: every matrix is a rotation, a stretch along perpendicular axes, and another rotation. Written as a sum of pieces, that decomposition says something even more useful. Each singular triple — a left vector, a singular value, a right vector — contributes one rank-one layer:
The layers are ordered. The first one, σ₁u₁v₁ᵀ, is the biggest single pattern in the matrix. The second adds the biggest pattern that is orthogonal to the first, and so on. If the singular values fall off quickly, then after a few terms the sum is already almost the whole matrix. Keeping the first k terms defines the rank-k truncation:
The theorem that makes this more than a heuristic is Eckart–Young: among all matrices of rank at most k, A_k minimises the error, and the leftover error is exactly the energy in the discarded singular values:
That one line is why nobody has to search for the best low-rank approximation: read the singular values, keep the biggest ones, stop. The next demo lets you watch it happen on a real image.
It is worth being precise about what "closest" means, because the choice of ruler is what makes the theorem true. The error above is the Frobenius norm, the square root of the sum of the squared entries — the matrix analogue of ordinary Euclidean length. Truncation is optimal for that ruler and for the spectral norm, but not for every conceivable notion of error. When you use the singular values as a ranking of importance, you are implicitly measuring energy in this entrywise-squared sense, and that is the sense in which the tail "is" the error: every discarded layer contributes its σ_i² and nothing else. It is also why truncation is a form of denoising. If the small singular values are mostly noise, dropping them does not merely compress the matrix, it removes the part of the matrix that was never signal to begin with.
Rank-k reconstruction of an image
A matrix is a picture; truncating it is compression
Treat a grayscale image as a matrix: one entry per pixel. The image below is generated on a canvas when the page loads — a smooth gradient with a few shapes on top — and its pixels are read into a 64×64 matrix. The SVD is computed once. The slider then rebuilds the image from only the first k singular triples, using the same cached U, Σ and V every time, so moving it is instant.
Drag the slider to change k. At small k you see a blurry sketch of the large shapes; the fine detail arrives only as the small singular values are let back in.
Two things are worth noticing. First, the first few layers already capture the broad light–dark structure, because a smooth gradient is nearly rank one and the large shapes are nearly rank a few. Fine texture and sharp edges are where the tail of the spectrum lives, and they are the last thing to come back. Second, the storage readout compares the full matrix, m·n numbers, with the truncated factors, k·(m+n) numbers — and at small k the factors are genuinely smaller. This is the mechanism behind every image and embedding compressor: keep the top layers, store the factors instead of the entries.
The arithmetic of that trade is the whole business case. Storing the full matrix costs mn numbers; storing the truncation costs k(m+n) plus the k singular values. The factors win whenever k(m+n) < mn, which for a square matrix means roughly k < n/2, and for a very tall matrix means an even smaller k. A recommender over millions of users and items is exactly that tall case, which is why a rank of a few dozen can replace a table the size of a small database. The cost is that you never get the original entries back — only their best low-rank shadow — so the question is always how much of the tail the application can tolerate.
As k rises, the reconstruction passes through a sequence of increasingly faithful pictures and becomes exact the moment k reaches the rank of the matrix. So the slider is really a dial between two extremes: a rank-one billboard of the dominant pattern, and the original matrix in full. Somewhere in between sits a rank that looks almost identical to the original but costs a fraction of the storage, and finding it is the skill. Notice that the slider never touches the expensive part. The decomposition was computed once at load; every position is just a weighted sum of cached columns, which is precisely why a trained model can apply a low-rank update at inference time without ever factorising a matrix on the fly.
Apply it to data and it becomes PCA
The top right singular vectors are the directions of greatest variance
Now put the points in the matrix instead of the pixels. Take n data points in d dimensions, subtract the mean so the cloud is centred, and stack them as the rows of an n×d data matrix X. Run the SVD, X = U\Sigma V^{\top}. The right singular vectors — the columns of V — point along the cloud's directions of greatest spread, and the singular values measure how much spread is in each. Equivalently, the covariance matrix
has eigenvectors V and eigenvalues σ_i²/n. So PCA is the SVD of a centered data matrix: the principal components are the top right singular vectors, and the variance they explain is the squared singular value divided by the total. The demo below uses the series' running point cloud. The two arrows are the principal axes, scaled by the standard deviations; the slider blends between the full cloud and its rank-one approximation, the projection onto the first principal component.
The cloud is seeded, so it is the same on every reload. At the left of the slider every point collapses onto the first principal axis; at the right you see the original cloud.
Watch what the rank-one approximation preserves. It keeps the spread along the long axis and throws away the spread across it. If the cloud is genuinely elongated, that second direction is small and discarding it costs almost nothing — you have reduced two coordinates to one while retaining most of the variance. That is dimensionality reduction in its entirety, and it generalises: replace two dimensions with hundreds, and the same test decides how many principal components are worth keeping.
Two details make the difference between PCA that works and PCA that misleads. The first is centering: the covariance formula only measures spread about the mean, so if you forget to subtract it the largest principal direction will often just point from the origin to the middle of the cloud — an artefact of where the data sits, not of how it varies. The second is units: if one column is measured in metres and another in millimetres, the covariance is dominated by the larger numbers and the leading component simply names the column with the biggest scale. When that happens you standardise each column to unit variance first, which turns PCA of the covariance into PCA of the correlation matrix. Neither step is optional bookkeeping; both change the answer, so it is worth being explicit about which one you did.
The reason this is an eigenproblem at all is that the covariance matrix is symmetric, and Part 15 proved that a symmetric matrix always has real eigenvalues and perpendicular eigenvectors. So the principal directions are not merely uncorrelated; they are orthogonal, and you can rotate the whole cloud to line them up with the coordinate axes without distorting it. In those coordinates the covariance is diagonal — each new axis carries an independent amount of variance — and sorting the axes by eigenvalue is the same as sorting the singular values of the data. That equivalence is the bridge between this part and the one before it: PCA does not need a separate theory, only the SVD applied to a matrix whose rows are data points.
Why k is usually tiny
The spectrum tells you how much structure there is
Everything depends on how fast the singular values decay. Plot each one's share of the total energy, σ_i² / Σ_j σ_j², and the shape of that curve decides your k. A steep curve means a few components do almost all the work; a flat curve means the matrix really is high-rank and truncation will hurt. The chart below uses the singular values of the image from the demo above, so you can see the same matrix from two angles.
Each bar is one singular triple's share of the total energy; the dashed line is the cumulative share. Where it crosses 90% is roughly the k a compressor would pick.
Natural images, embeddings and recommender tables all tend to show a steep initial drop followed by a long, low tail. The steep part is structure you want; the tail is detail, noise, or redundancy you can afford to lose. That is why a matrix with millions of entries is often well described by a few hundred numbers, and why "low-rank" is not a restriction you impose but a property you discover.
Choosing k in practice is a judgement made from this curve. The old-fashioned recipe is to look for an elbow — the index where the bars fall off a cliff and flatten out — and to cut just after it. A more principled version sets a budget: pick the smallest k whose cumulative share crosses a threshold such as 90% or 99%, which is exactly what the readout reports. Both are heuristics, and both can be fooled. A slowly decaying spectrum has no elbow, and a single huge spike can hide a lot of structure in the remaining components, so treat the curve as evidence rather than as an oracle. The honest test is downstream: reconstruct, measure the error that actually matters for your task, and decide whether the compression was worth it.
There is also a numerical reason to like truncation, which Part 19 develops in full. The condition number of a matrix is the ratio σ₁/σ_n, so the smallest singular values are exactly the ones that make a system sensitive to noise. Dropping them does not just save storage; it regularises the problem, because the directions you removed were the ones where a small perturbation of the data produced a large change in the answer. This is why truncated SVD appears inside solvers and inverse problems under the name regularisation: you deliberately trade a little bias for a large reduction in variance. The discarded tail is where instability lives, and truncation is the simplest way to lock it out.
Where this shows up
One decomposition, two worlds
Embeddings, recommenders and LoRA
A user–item rating table is low-rank because tastes are few, and factorising it is recommender systems. Word and token embeddings are compressed the same way. Most strikingly, LoRA fine-tunes a large model by freezing the weights and learning a rank-k update, a product of two skinny matrices — the exact structure this part builds. See scaling and training at size for where those updates fit.
Low-dimensional structure in a point cloud
A laser scan of a wall, a set of matched image features, or a reconstructed set of 3-D points rarely fills space; it lies near a plane or a line. PCA finds that structure, and the small singular values expose the noise. Structure from motion leans on exactly this: a rigid scene's tracked points form a low-rank measurement matrix, and the rank reveals the motion.
Takeaways
- The truncated SVD A_k = Σ_{i≤k} σ_i u_i v_iᵀ is the best rank-k approximation, and the residual energy is the sum of the discarded σ_i².
- Each singular triple is a rank-one layer; the ordering by σ is a ranking of importance.
- PCA is the SVD of a centered data matrix: principal directions are the right singular vectors, and explained variance is σ_i² normalised by the total.
- How far you can truncate is a property of the data, read directly off the decay of the spectrum.
Further reading
- Grant Sanderson, "The SVD" (and the companion chapters on eigendecomposition), Essence of Linear Algebra, 3Blue1Brown.
- Gilbert Strang, 18.06 Linear Algebra, MIT OpenCourseWare — Lectures 29 and 30 on singular value decomposition and applications.
- Jonathon Shlens, "A Tutorial on Principal Component Analysis" — the careful derivation of PCA as an eigenproblem and as an SVD.
- Carl Eckart and Gale Young, "The approximation of one matrix by another of lower rank," Psychometrika, 1936 — the theorem behind the slider.