Calculus of variations & optimal control

The unknown is a function, not a point

Until now the unknown has been a point: a scalar $x$, a vector, a matrix. A trajectory is a different animal. Choosing a path means choosing a value $y(t)$ at every instant, so the unknown is a function and the thing being scored is computed from the whole function. An objective that eats a function and returns a number is called a functional, and the branch of calculus built on it is the calculus of variations.

Our running example is the two-link planar arm. Its end-effector must leave rest at $A$ and arrive at rest at $B$ in exactly $T$ seconds. “From rest” means zero velocity and zero acceleration at both ends. Many curves satisfy that; the smoothest is the one with least rate of change of acceleration, the jerk. Writing $\tau = t/T$ for the normalised time, the winner is the minimum-jerk quintic

It is the unique quintic with $\sigma(0)=0$, $\sigma(1)=1$ and $\sigma'=\sigma''=0$ at both ends, and its jerk cost is the textbook value below — a number you will read off the canvas and check against the analytic one.

Drag $A$ and $B$, switch the optional via point on to force a bend, and vary the duration. The top canvas is the workspace; the strip below is the scalar progress, speed and acceleration of the two candidates — the straight constant-speed line versus the minimum-jerk quintic. The straight line does no jerking inside, but it starts and stops instantaneously, so all of its roughness hides in two instants; the quintic spreads it out. On the same finite-difference grid its cost is orders of magnitude larger.

A functional, and its derivative with respect to a function

A functional of a scalar path has the form

where $L$ is the Lagrangian — the local price of position and slope at each instant. Distance, time, energy and jerk cost all fit this template. To find the best $y$ we perturb the entire curve by a tiny multiple of another curve, $y_\varepsilon(t) = y(t) + \varepsilon\,\eta(t)$, with $\eta$ vanishing at the endpoints so the ends stay put. Now $J(\varepsilon)$ is an ordinary function of the single number $\varepsilon$, and an optimum must satisfy

That quantity is the first variation of $J$: literally a derivative taken with respect to a function. The demo makes it visible. Start from the minimum-jerk path, bend it by $\alpha$ times a fixed bump $b(t)=64\,t^3(1-t)^3$ that vanishes to third order at both ends, and plot the cost $J(\alpha)$. It is a parabola whose bottom sits exactly at $\alpha = 0$ — the derivative with respect to the shape is zero at the optimum, just as an ordinary derivative is zero at a minimum.

Setting the first variation to zero, once and for all

Do the perturbation calculation symbolically. Substitute $y_\varepsilon$ into the functional and differentiate under the integral:

The second term still contains $\eta'$, which is awkward, so integrate it by parts:

The boundary term dies because $\eta(a)=\eta(b)=0$. What is left must vanish for every bump $\eta$, and the only way an integral against an arbitrary function can be zero is if the bracket itself is zero at every point. That gives the Euler–Lagrange equation:

For the shortest path between two points, $L=\sqrt{1+y'^2}$. Since $L$ does not mention $y$, the equation reduces to $\frac{d}{dt}\big(y'/\sqrt{1+y'^2}\big)=0$, i.e. $y'$ is constant: the straight line. Because this $L$ has no explicit $t$, the Beltrami identity $y'\frac{\partial L}{\partial y'}-L=\text{const}$ holds too, and for this $L$ it says the same thing. The demo takes the straight line and perturbs it by $\varepsilon\sin(\pi t)$; the readout shows $dJ/d\varepsilon$ and the residual of the Euler–Lagrange condition. At the minimiser both are $\approx 0$.

Choosing a function $u(t)$, not just a path

Optimal control is the same idea with a dynamic constraint. The plan is the input $u(t)$ and the state obeys $x''=u$ — a point mass where $u$ is acceleration (a double integrator). Pick the input that carries the mass from $x(0)=0,\,v(0)=0$ to $x(T)=1,\,v(T)=0$ with the least control energy

The state is a cubic in $t$ (because $x''''=0$), the control is its second derivative, and imposing the four endpoint conditions gives the optimum directly. For $T=1$:

The optimal control is linear in time, $J^\star=\int_0^1(6-12t)^2dt=12$. The slider adds a smooth perturbation $p\,g(t)$ to the state with $g=16t^2(1-t)^2$ — value and slope both zero at the ends, so the boundary conditions stay satisfied. The cost is $J(p)=12+204.8\,p^2$: a bowl with its minimum, and a flat optimal control, at $p=0$. This is the linear-quadratic regulator in miniature; the general construction adds a costate and the Pontryagin minimum principle, but the geometry is what the canvas shows.

A computer cannot store a function

The theory lives in continuous time, but a solver must store finitely many numbers. The universal move is to sample the function and replace the integral by a sum. For the arc-length functional that means an inscribed polyline:

This is exactly the Riemann sum of Volume I, Part 8, applied to a functional instead of an area: the polyline is short because it cuts the corners of the curve, and the error falls like $1/n^2$. The waypoint slider coarsens and refines the discretisation; the bottom canvas plots the error on log–log axes, where the slope of the line is the order of convergence. Every trajectory optimiser in practice works this way: discretise, then hand the resulting finite-dimensional problem to the methods of Part 10 or Nonlinear Optimization.

Trajectories are the job description

Robot navigation is trajectory optimisation with obstacles attached: plan a path that is smooth, dynamically feasible and cheap — the same functional as this page, with a constraint added. In state estimation the same functional reappears as a sum of squared errors over a whole trajectory — see Pose Graphs & Loop Closure, where the “path” is the robot's history and the cost is the residual of every odometry and loop-closure measurement. The dynamic constraint $x''=u$ is an ODE, the subject of Differential equations, and the accumulation that turns a rate into a functional is the Fundamental Theorem of The Fundamental Theorem. Minimum-jerk tracking is also why industrial arms move along quintics, and model-predictive control re-solves a short-horizon version of this page at every control tick.

• Wikipedia, Calculus of variations — the first variation and the Euler–Lagrange equation in full generality.
• Wikipedia, Euler–Lagrange equation — worked examples, Beltrami's identity and the boundary-term argument repeated here.
• Wikipedia, Optimal control — the Pontryagin minimum principle and the linear-quadratic regulator the double integrator previews.