Epipolar Geometry

Setup

Say camera 1 sees a point at pixel x₁. That pixel corresponds to an entire ray in 3D — the point could be anywhere along it (exactly the ambiguity from Part 4). Now project that whole ray into camera 2's image. Because a ray is a 3D line, its image under a second pinhole camera is also a straight line — the epipolar line. Wherever the true 3D point actually is, its image in camera 2 must fall somewhere on that line. That's the entire idea; the rest of this page is just making it precise and fast to compute.

Foundations

The two camera centers and a 3D point X define a plane — the epipolar plane. It slices each image plane along a line: the epipolar line. Every epipolar line in image 2 passes through one fixed point, the epipole e₂ — the image, in camera 2, of camera 1's own center (and vice versa for e₁ in image 1).

For calibrated cameras (points expressed in normalized camera coordinates, intrinsics already divided out), the relationship between the two cameras' relative rotation R and translation t is packed into the 3×3 essential matrix:

and the constraint that a correspondence (x₁, x₂) must satisfy is beautifully simple:

For raw pixel coordinates (intrinsics not divided out — the usual case with an uncalibrated or unknown camera), the same idea generalizes to the fundamental matrix F = K₂⁻ᵀEK₁⁻¹, with the identical-looking constraint:

Fx₁ is literally the epipolar line in image 2, in the line-equation form au + bv + c = 0. Given enough correspondences (8 is the classic minimum — the eight-point algorithm solves the same kind of linear system you'll see used for homographies in the next part), F can be estimated directly from two uncalibrated images, with no camera calibration required at all.

Interactive

Click anywhere in the left image to choose a pixel in camera 1. That fixes a 3D ray. The right image immediately draws its epipolar line — computed from F, using only the two camera poses, before we've even said how far along the ray the point is. Then drag the depth slider: the true 3D point moves along the ray, its actual projection in image 2 (the dot) slides, but always along the epipolar line (the dashed line).

Now move camera 2's sliders instead: the epipole and every epipolar line move with it. Notice that whatever pixel you clicked in image 1, its epipolar line in image 2 always passes through the same epipole — try clicking several different points and watch all their lines fan out from one spot.

Derivation

Step 1 handed you E = [t]×R as a fact. Here's where it comes from. Put camera 1 at the origin with identity orientation, and camera 2 at pose (R, t) relative to it (t is the direction from camera 1 to camera 2, expressed in camera 1's frame). A 3D point X gives a ray direction x̃₁ in camera 1 and, once rotated into camera 2's frame, direction Rx̃₁; its actual observed direction in camera 2 is x̃₂.

Camera 1's center, camera 2's center, and X all sit in one plane — the epipolar plane from Step 1. Equivalently: the baseline vector t, the ray direction Rx̃₁ (camera 1's ray, expressed in camera 2's frame), and the ray direction x̃₂ are three vectors lying in that same plane, so they're coplanar. Three vectors a, b, c are coplanar exactly when their scalar triple product vanishes: a·(b×c) = 0. Applied here:

The cross product t × (·) is a linear map — it can be written as a matrix multiply. For t = (t₁, t₂, t₃), define the skew-symmetric matrix [t]× so that [t]× v = t × v for any v:

Substituting turns the triple product into a pure matrix expression, and that's the whole derivation:

Nothing more went into it than "three vectors in one plane have zero triple product." E is just the matrix form of "cross with t, then rotate by R."

Properties

Everything below is about the uncalibrated F = K₂⁻ᵀEK₁⁻¹, since that's what you actually estimate from pixel correspondences with no calibration in hand.

Rank 2, always. [t]× has rank 2, not 3: its null space is t itself, since t × t = 0. Multiplying by the full-rank rotation R doesn't change rank, so E = [t]×R also has rank exactly 2. Multiplying by the invertible calibration matrices K₂⁻ᵀ and K₁⁻¹ preserves rank too, so F is rank 2 as well — det(F) = 0, always, for any real stereo pair.

7 degrees of freedom. A 3×3 matrix has 9 entries. F is only defined up to an overall scale (the constraint x̃₂ᵀFx̃₁=0 is homogeneous in F) — that's 1 DOF gone. The rank-2 constraint det(F)=0 removes 1 more. 9 − 1 − 1 = 7. This is exactly why the minimal correspondence count for uncalibrated F is 7, not 9 — see the seven-point algorithm below.

Epipoles are F's null vectors. The epipolar line in image 1 corresponding to a point x̃₂ is l₁ = Fᵀx̃₂, and every such line passes through e₁ — by definition, the epipole is where the baseline itself lands in image 1, and every epipolar plane contains the baseline, so every epipolar line contains its image. So e₁ᵀl₁ = 0 for every x̃₂, i.e. e₁ᵀFᵀx̃₂ = 0 for all x̃₂, which forces e₁ᵀFᵀ = 0, i.e.:

The epipoles are the right and left null vectors of F — exactly the kind of "smallest singular vector" object the eight-point algorithm below computes anyway, so once you have F the epipoles are almost free.

Order matters, transposed. F₁₂ (image 1 → image 2 convention, x̃₂ᵀF₁₂x̃₁=0) and F₂₁ (the same pair, roles swapped, x̃₁ᵀF₂₁x̃₂=0) are related by a plain transpose: F₂₁ = F₁₂ᵀ. Transposing the scalar equation x̃₂ᵀF₁₂x̃₁=0 (a 1×1 matrix, equal to its own transpose) gives x̃₁ᵀF₁₂ᵀx̃₂=0, which is exactly the swapped-order constraint with F₂₁=F₁₂ᵀ.

Properties, continued

The two structural facts from the previous step — F is only defined up to scale, and det(F)=0 — together explain why a fundamental matrix is not an arbitrary 3×3 matrix. Nine numbers, but only seven degrees of freedom. It is worth watching each of the two extra dimensions come off in turn.

(a) Overall scale is meaningless — 9 → 8. The constraint x̃₂ᵀFx̃₁=0 still holds if every entry of F is multiplied by the same nonzero number, so F and λF describe exactly the same epipolar geometry. Fixing that arbitrary gauge — say by requiring a unit-norm f — spends one of the nine numbers. Eight remain.

(b) det(F)=0 — 8 → 7. A real fundamental matrix has rank 2, so its smallest singular value is zero. Unlike the scale choice, this is a genuine nonlinear constraint, not a gauge. The plain linear eight-point solve enforces only (a): it returns the unit-norm f minimizing ‖Af‖, which for noisy correspondences reshapes to a full-rank 3×3 matrix with det(F)≠0 — near the rank-2 surface but not on it. Snapping it back is the same SVD trick used for homographies: factor F=U·diag(σ₁,σ₂,σ₃)·Vᵀ, set σ₃=0, and reassemble. By Eckart–Young that is the closest rank-2 matrix in Frobenius norm, and it is exactly what a constrained or nonlinear fit converges to.

Interactive derivation

Every correspondence (x̃₁, x̃₂) gives one linear equation in the 9 unknown entries of F. Writing x̃₁=(x₁,y₁,1), x̃₂=(x₂,y₂,1) and expanding x̃₂ᵀFx̃₁=0 term by term:

That's just a·f = 0 for the 9-vector f = (f₁₁,f₁₂,...,f₃₃) and the row a = (x₁x₂, y₁x₂, x₂, x₁y₂, y₁y₂, y₂, x₁, y₁, 1) — the outer product of x̃₁ and x̃₂, flattened. Stack one such row per correspondence into a matrix A: N correspondences give an N×9 system Af = 0. With exactly 8 correspondences in general position, the null space of A is (generically) one-dimensional — that's the "eight" in eight-point. With more than 8 (and real pixel noise), there's usually no exact solution, so instead you solve the homogeneous least-squares problem: minimize ‖Af‖ subject to ‖f‖=1. That minimizer is the right singular vector of A belonging to its smallest singular value — equivalently, the eigenvector of AᵀA (a 9×9 matrix you can actually build by hand) with the smallest eigenvalue. Reshape that 9-vector back into a 3×3 matrix and you have a candidate F.

One more wrinkle: the SVD solution for F generically comes out full rank (3), because real measurement noise pulls it off the rank-2 manifold that a true fundamental matrix must live on. The fix is to take the SVD of the resulting F = U·diag(σ₁,σ₂,σ₃)·Vᵀ and zero out the smallest singular value: F₂ = U·diag(σ₁,σ₂,0)·Vᵀ. By the Eckart–Young theorem, this F₂ is provably the closest rank-2 matrix to the noisy F in Frobenius norm — the cheapest possible way to snap an almost-valid matrix back onto the constraint surface it's supposed to satisfy.

Minimal solvers

Seven-point algorithm. With only 7 correspondences, A is 7×9 and its null space is generically two-dimensional, spanned by matrices F₀ and F₁ (found the same way — the two smallest-eigenvalue eigenvectors of AᵀA). Every rank-2 F consistent with the 7 points has the form F = F₀ + λF₁ for some scalar λ, so the rank-2 constraint det(F₀+λF₁) = 0 becomes a cubic polynomial in λ. A cubic has 1 or 3 real roots, so seven-point produces 1 to 3 candidate matrices; an 8th correspondence (or a cheirality/reprojection check) picks the right one. It exists purely because 7 points aren't enough to pin down the 2 DOF a 9-entry, scale-free, rank-2 matrix has left over — you get a 1-parameter family instead of a point.

Five-point algorithm (calibrated). If the cameras are calibrated, you're solving for E, not F, and E carries one more constraint than "rank 2": its two nonzero singular values must be equal. That's 2 extra constraints beyond what F has, dropping the minimal correspondence count from 7 to 5. Nistér's five-point algorithm (2004) solves this minimal, more constrained problem directly — it's algebraically more involved (a polynomial system solved via Gröbner bases or resultants) and this page won't derive it, but it's worth knowing it exists: it's what nearly every modern visual-odometry and structure-from-motion pipeline actually calls, because the extra constraint makes it far less prone to the planar-scene degeneracy that eight-point suffers from below, and using fewer points means fewer that need to be outlier-free inside a RANSAC loop.

Degeneracies

Planar scenes. If every 3D point lies on one plane, the correspondences are secretly related by a single homography (Part 8), and there's more than one rank-2 F consistent with them — the eight-point system becomes ill-posed, with a poorly separated null space instead of one clean answer. In practice, this is exactly why real pipelines don't commit to F blindly: they fit both a homography H and a fundamental matrix F to the same correspondences, then use a model-selection criterion (GRIC is the classic one) that scores each fit while penalizing H's much smaller degree of freedom count, and keep whichever model actually explains the data — a near-planar scene should win on H, a general scene on F.

Pure rotation. If the camera only rotates in place (t = 0) — think of standing still and turning your phone — then E = [t]×R with [0]× = 0, so E is identically the zero matrix, not merely rank-2. There's no baseline, hence no epipolar plane, hence literally zero triangulation information in the geometry: any correspondence satisfies x̃₂ᵀ·0·x̃₁=0 trivially, so the constraint carries no information at all. This is why "just rotate the phone in place" never works for stereo or SfM, no matter how many frames you capture.

Points on the baseline. A 3D point sitting exactly on the line through both camera centers projects to the epipole itself in both images — its epipolar plane degenerates to a pencil (any plane through the baseline satisfies the coplanarity condition), so its epipolar line is undefined. In practice this shows up as instability for points near the epipole, not just exactly on it: the closer a correspondence is to the epipole, the less its epipolar line constrains anything.

Calibration changes how badly the planar case bites. The 5-point algorithm gets a full worked treatment later in this series, in Part 13; for now, the side-by-side demo below fits both models to the same near-planar scene. Drag the planarity slider from a general 3D cloud toward a coplanar one and watch the uncalibrated eight-point system lose its well-separated null space, while the calibrated essential matrix stays pinned to the equal-singular-value manifold, where rank 2 and det(E)=0 hold by construction.

The configurations that break these estimators — planar scenes, pure rotation, critical surfaces, near-degenerate baselines — are gathered and demonstrated in Part 18.

Epipolar geometry constrains where a match can be, but the payoff — dense, pixel-by-pixel stereo matching — needs the epipolar lines turned into simple horizontal scanlines first. That rectification-and-disparity pipeline gets its own page: Part 9.

Recap