PnP: Absolute Pose from 2D–3D Correspondences

Setup

You have N 3D points X₁, …, Xₙ whose world coordinates are already known — they came from a prior triangulation, a scan, or an earlier reconstruction. You have one new image, and for at least some of those points you know which pixel x₁, …, xₙ each one landed on — the matching itself isn't this page's job, only what to do once it's done. You also know the camera's calibration K (Part 5). What you don't know is the camera's pose: the rotation R and position it was sitting at when it took this photo. PnP is the problem of finding the (R, t) that makes all of these consistent at once:

Foundations

The unknown here is the 3×4 camera matrix P = K[R | t] — 12 numbers, but only 11 independent (it's defined up to an overall scale). Exactly the same cross-product idea used for homography DLT (Part 8) and the 8-point algorithm (Part 6, epipolar) applies directly: since x̃ₓ and PX̃ₓ point the same direction, their cross product vanishes.

Writing P's three rows as 4-vectors p₁ᵀ, p₂ᵀ, p₃ᵀ stacked into one 12-vector p, and xₓ = (xₓ, yₓ) for the observed pixel, two of the three cross-product components are independent and linear in p:

Two rows per correspondence, 12 unknowns up to scale (11 DOF) — so 6 correspondences (12 equations) pin it down exactly; more than 6 gives an overdetermined, noise-averaging system. Stack every correspondence's row pair into one matrix A and the whole problem collapses to the same homogeneous least-squares shape this series keeps returning to — the 8-point algorithm's Af = 0, the homography DLT's Ah = 0, and now:

Reshape the resulting p back into the 3×4 matrix P. Because K is already known here (unlike Part 4's calibration problem, where K is what RQ-decomposition extracts from P's leading 3×3 block), the same decomposition runs in reverse: use K to strip the calibration out of P instead of reading it off. In practice it's simplest to normalize the pixels first — multiply every observation by K⁻¹ to get calibrated rays — so the DLT solves directly for [R | t] (up to scale), and only R needs cleaning up:

Interactive

14 known 3D points (think: a small patch of an existing reconstruction) are observed by one new camera. Pixel noise and a few injected outliers (mismatches) are added to the observations before the solver ever sees them — exactly the situation a real resection step is in.

With zero noise the raw DLT solve is already essentially exact — there's nothing to refine. Push noise up and the raw estimate drifts measurably; refinement (a handful of Gauss–Newton steps minimizing true reprojection error, the same machinery as Part 10's nonlinear triangulation) pulls it most of the way back. Add outliers, though, and refinement stops helping — a few wildly wrong correspondences drag the least-squares solve off no matter how well you polish it afterward. That's the motivation for the RANSAC demo below.

Foundations

DLT needs 6 points. But given K, the true minimum is 3 — and getting there means leaving linear algebra behind for plain geometry. Three known 3D points X₁, X₂, X₃ and the (unknown) camera center C form a tetrahedron. The three known inter-point distances dₓᵀ (from the 3D points themselves) are fixed; the three angles θₓᵀ between the viewing rays C→Xₓ and C→Xᵀ are also known — they come straight out of K and the observed pixels, with no camera pose needed at all, since an angle between two rays through the same center doesn't depend on where that center is. What's unknown are the three distances from the camera to each point, d₁ = |CX₁|, d₂, d₃. The law of cosines in the triangle C, X₁, X₂ ties them together:

The same relation holds for the (1,3) and (2,3) pairs, giving three simultaneous quadratics in d₁, d₂, d₃ — known coefficients (θₓᵀ, dₓᵀ), three unknowns. Eliminating variables between the three equations produces a single quartic polynomial in one of them, which has up to 4 real roots — so P3P alone hands back up to 4 algebraically valid camera poses, all consistent with the same three rays and three distances.

Foundations

P3P is minimal but ambiguous; DLT is unambiguous but its cost grows with N unknowns tied one-to-one to the points. EPnP is the standard middle path used by real systems (it's what OpenCV's solvePnP reaches for by default, and what runs inside solvePnPRansac's minimal-sample loop for larger samples). The key idea: instead of treating each of the N 3D points as its own unknown, pick just 4 virtual control points once, and express every real 3D point as a fixed weighted combination of those 4 (barycentric-style coordinates, computed once from the known Xₓ and frozen for the rest of the solve). The problem of "where is every one of these N points, really, in the camera's frame" becomes "where are these 4 control points" — 12 unknowns total, regardless of how large N is. That turns what would otherwise be an N-scaling problem into a small, fixed-size linear system solved once per image, which is exactly why EPnP is the fast, accurate default for N ≥ 4 points in production structure-from-motion and SLAM pipelines.

Foundations

Real correspondences come from feature matching, and feature matching produces outliers — wrong matches that no amount of least-squares polishing will fix, exactly as the demo above showed. The fix is the same one Part 7 used for the 8-point and homography DLT: wrap the minimal solver in RANSAC. Draw a minimal sample (3 points for P3P, plus a 4th to disambiguate its candidates — or, as this page's demo does, 6 points for a minimal DLT solve, which needs no disambiguation step but is otherwise the identical loop), solve for a candidate pose, count how many of the other correspondences it reprojects within a small pixel threshold, and repeat. Whichever candidate wins the most inliers gets refined — typically with EPnP or a nonlinear reprojection-error minimization like the Gauss–Newton step above — on its full inlier set, and that refined pose is the camera's registered position.

Interactive

Same 14-point cloud and true camera as above, plus a controllable outlier fraction. "Naive DLT" resects on every correspondence at once. "PnP + RANSAC" repeatedly resects on random 6-point samples, scores each by reprojection inliers, and refines the winner on its inlier set.

Recap