Curvature and convergence
Why does gradient descent zig-zag, crawl, and then explode when you nudge the learning rate? The answer was settled the moment you wrote down the Hessian. This page is the why: shape a quadratic, race three optimizers on it, and read the convergence rate straight off the eigenvalues. The algorithms themselves — gradient descent, momentum, Adam, Newton, Gauss–Newton, Levenberg–Marquardt — are built in Nonlinear Optimization; here we only explain the landscape they are walking.
Shape the surface
Curvature is a matrix, not a number
Our running example is the two-link planar arm reaching for a target. Near the target, the squared reach error as a function of the two joint angles is, to second order, a quadratic. Choose axes along its principal directions and the whole landscape is two independent parabolas:
The Hessian $H$ is the local curvature matrix from Multivariable Taylor & critical points. Its eigenvalues are the curvatures along the principal directions, and the level sets of $f$ are ellipses:
Raise the condition-number slider and the ellipses stretch: the steep direction gets steeper while the shallow direction stays put. The magenta axis is the high-curvature eigendirection $\lambda_{\max}=\kappa$; the blue axis is the low-curvature one $\lambda_{\min}=1$. The ratio $\kappa=\lambda_{\max}/\lambda_{\min}$ is the page's whole story.
Level ellipses of the quadratic, with the two Hessian eigen-axes drawn through the minimum at the origin.
A race on one surface
Three optimizers, one landscape
Write the iterate as a sum of eigenvectors, $x_k = a_k \mathbf v_1 + b_k \mathbf v_2$. Gradient descent with step $\eta$ acts on each coordinate independently:
So every method is a contraction in the eigenbasis, and the spectrum $\{\lambda_i\}$ sets its rate. Momentum adds a velocity term that cancels the alternating sign of the steep mode; Adam rescales each coordinate by a running estimate of its gradient magnitude. Those are the algorithms of Nonlinear Optimization — here we only watch them race. Drag the start point, press Play, and read the iteration at which each one reaches $|f-f^\star| < 10^{-4}$ on the log-log plot below.
Top: the three trajectories on the contours; drag the white start dot. Bottom: the same runs on log-log axes — a straight line is geometric (linear) convergence.
Why steepest descent zig-zags
The step is capped by λ_max, the progress by λ_min
Stability requires the step to satisfy $\eta < 2/\lambda_{\max}$, so the largest curvature decides how far you may step. But the smallest curvature decides how fast you finish, because its factor $1-\eta\lambda_{\min}$ is closest to one. With the best fixed step, $\eta = 2/(\lambda_{\max}+\lambda_{\min})$, both coordinates decay by the same ratio
The steep coordinate cannot decay monotonically at that step — it flips sign every iteration, decaying by $r$ each time. That is the zig-zag: large oscillations across the narrow valley while the valley floor is traversed at a crawl. The condition number is exactly the ratio that makes the two timescales disagree. Increase κ and count the reversals.
Steepest descent from a fixed start. The dashed blue line is the valley floor (the $\lambda_{\min}$ direction); the path bounces across it.
How big a step?
The contraction factor is |1 − ηλ| per eigenvalue
Fix the Hessian and sweep the learning rate. Each eigendirection is contracted by $|1-\eta\lambda_i|$: below one it decays, above one it grows, and near one it stalls. The worst eigenvalue gives the worst factor, so the per-iteration contraction is $\max_i |1-\eta\lambda_i|$. For a fixed spectrum the useful window is
Slide η from a crawl through the sweet spot into divergence. The readout prints both contraction factors and the verdict. This is the only knob the algorithms share; how each one chooses it belongs to Nonlinear Optimization.
Gradient descent at the chosen step, with the fixed spectrum $\lambda_{\max}=10$, $\lambda_{\min}=1$. The path leaves the frame when η exceeds $2/\lambda_{\max}$.
Curvature is the fix
One Newton step versus many gradient steps
Gradient descent has to guess a single step for a whole spectrum. Newton's method uses the curvature itself. Minimising the quadratic model $f + \mathbf g^{\mathsf T}p + \tfrac{1}{2}p^{\mathsf T}H p$ gives $\mathbf g + H p = 0$, so the Newton step is the solve
On a quadratic the model is exact, so one step lands on the minimum — regardless of κ. The demo below solves $Hp=-\mathbf g$ with the same Gaussian elimination a Gauss–Newton or Levenberg–Marquardt step uses; slide κ and watch Newton keep its single step while gradient descent needs ever more. Cost per step is the trade: Newton forms and solves an $n\times n$ system, which is $O(n^3)$ and infeasible when the parameter count is large.
Gradient descent (blue, many small steps) and the single Newton arrow (magenta) to the minimum at the origin.
Methods at a glance
What curvature costs, and what it buys
For the current condition number $\kappa$, the iteration counts below are the ones the race actually runs to. Directional curvature is what separates them: gradient descent sees one number, momentum remembers the last direction, Newton reads the whole matrix. Their construction and their many variants are the subject of Nonlinear Optimization.
| Method | Cost per step | Iterations to |f − f*| < 10⁻⁴ | Condition-number dependence |
|---|---|---|---|
| Gradient descent | $O(n)$ — one gradient | — at κ = — | Rate $((\kappa-1)/(\kappa+1))^{2k}$; degrades fast as κ grows |
| Momentum (heavy ball) | $O(n)$ — gradient + velocity | — at κ = — | Same spectrum, faster slow mode; still κ-limited |
| Newton | $O(n^3)$ — solve $Hp=-g$ | — at κ = — | Invariant to κ on a quadratic; one exact step |
Where this shows up
The why behind the solvers
This page deliberately stopped at the explanation. The algorithms live in Nonlinear Optimization, and every one of them is a response to the picture above: Gradient Descent takes the step this page showed can only crawl in the shallow direction; Newton replaces it with the $H^{-1}$ solve from Step 5; Gauss–Newton builds an approximate Hessian $J^{\mathsf T}J$ when a true one is unavailable; and Levenberg–Marquardt interpolates between the two, damping exactly the ill-conditioned directions that make κ large. The curvature machinery itself — eigenvalues, the quadratic model, the second-derivative test — is Multivariable Taylor & critical points (Volume I, Part 14). And on the arm: the reach error is a surface whose conditioning changes with the pose, which is precisely why trust-region solvers watch it.
Notation to carry forward
| Object | Reads as | What it decides |
|---|---|---|
H = ∇²f | The Hessian, the curvature matrix | The shape of the local quadratic; bowl, dome or saddle |
λ₁ ≥ λ₂ ≥ … | Eigenvalues of H | Curvature along each principal direction |
κ = λ_max / λ_min | Condition number | How mismatched the timescales are; sets the slow mode |
|1 − ηλ_i| | Contraction factor for eigendirection i | < 1 decays, > 1 diverges; > 2/λ_max is unstable |
η* = 2/(λ_max+λ_min) | Best fixed gradient step | Gives the rate ((κ−1)/(κ+1)) per step |
p = −H⁻¹g | The Newton step | Reaches the quadratic's minimum in one move |
f(x_k) − f* | Suboptimality | The vertical axis of the log-log convergence plot |
Further reading
- Boyd & Vandenberghe, Convex Optimization — Chapters 9 and 10 derive the condition-number bound and the subgradient machinery from scratch.
- Nocedal & Wright, Numerical Optimization — the convergence-rate theorems and Newton, Gauss–Newton and trust-region methods in full.
- Goodfellow, Bengio & Courville, Deep Learning — Chapter 8 covers momentum and Adam, the methods this page only raced.