
Lecture 12
Computer Science Department, Colorado School of Mines
By the end of this lecture, you should be able to:

\[ \underbrace{\lambda}_{\text{depth } Z_c}\, \underbrace{\mathbf{x}}_{\substack{\text{pixel}\\ (u,\,v,\,1)}} = \overbrace{ \underbrace{\color{#c53030}{\mathbf{K}}}_{\substack{\text{intrinsic}\\ \text{matrix}}}\; \underbrace{\big[\,{}^{c}\mathbf{R}_{w} \mid {}^{c}\mathbf{t}_{w}\,\big]}_{\substack{\text{extrinsic}\\ \text{matrix}}} }^{\mathbf{P}:\ \text{camera matrix}}\; \underbrace{{}^{w}\mathbf{X}}_{\substack{\text{world point}\\ (X,\,Y,\,Z,\,1)}} \]
\[\lambda\,\mathbf{x} = \mathbf{K}\,\big[\,{}^{c}\mathbf{R}_{w} \mid {}^{c}\mathbf{t}_{w}\,\big]\;{}^{w}\mathbf{X}\]
Which quantities are known and which are unknown defines the problem:
| Problem | Pixels \(\mathbf{x}\) | Intrinsics \(\mathbf{K}\) | Extrinsics \({}^{c}\mathbf{R}_{w},\ {}^{c}\mathbf{t}_{w}\) | 3D points \({}^{w}\mathbf{X}\) |
|---|---|---|---|---|
| Projection (simulate a camera) | unknown | known | known | known |
| Camera calibration | known | unknown | unknown | known |
| Two-view geometry | known | known | unknown | unknown |
| Triangulation | known | known | known | unknown |
Assume given an image of a target whose 3D points \({}^{w}\mathbf{X}_i\) are known, with their pixel positions \((u_i, v_i)\).
Unknown: the camera matrix \(\mathbf{P}\), with rows \(\mathbf{p}_1^{T}, \mathbf{p}_2^{T}, \mathbf{p}_3^{T}\).
From camera model \(\lambda\,\begin{bmatrix} u_i \\ v_i \\ 1 \end{bmatrix} = \begin{bmatrix} \mathbf{p}_1^{T} \\ \mathbf{p}_2^{T} \\ \mathbf{p}_3^{T} \end{bmatrix}\,{}^{w}\mathbf{X}_i,\) we have: \[\lambda u_i = \mathbf{p}_1^{T}\,{}^{w}\mathbf{X}_i, \qquad \lambda v_i = \mathbf{p}_2^{T}\,{}^{w}\mathbf{X}_i, \qquad \lambda = \mathbf{p}_3^{T}\,{}^{w}\mathbf{X}_i \]
Substituting \(\lambda=\mathbf{p}_3^{T}\,{}^{w}\mathbf{X}_i\) in the first two equations gives two equations per point, both linear in the 12 entries of \(\mathbf{P}\): \[\mathbf{p}_1^{T}\,{}^{w}\mathbf{X}_i - u_i\,\mathbf{p}_3^{T}\,{}^{w}\mathbf{X}_i = 0\] \[\mathbf{p}_2^{T}\,{}^{w}\mathbf{X}_i - v_i\,\mathbf{p}_3^{T}\,{}^{w}\mathbf{X}_i = 0\]

Example output for the target points of the previous slide:
\[\mathbf{P}_{\text{DLT}} = \begin{bmatrix} -0.487 & 0.221 & -0.202 & 0.483 \\ 0.108 & 0.0902 & -0.514 & 0.395 \\ -0.000563 & -0.000466 & -0.000631 & 0.00157 \end{bmatrix}\]
When we have more equations than unknowns (plus noise), there is no exact solution. Least-squares find the best approximate solution that minimizes the sum of squared errors. There are two cases:
Nonhomogeneous \(\mathbf{A}\mathbf{p} = \mathbf{b}\): \[\min_{\mathbf{p}} \|\mathbf{A}\mathbf{p} - \mathbf{b}\|^2 \;\Rightarrow\; \mathbf{p} = (\mathbf{A}^{T}\mathbf{A})^{-1}\mathbf{A}^{T}\mathbf{b}\]
Homogeneous \(\mathbf{A}\mathbf{p} = \mathbf{0}\):
Calibration is the homogeneous case: \(\mathbf{P}\) is only defined up to scale.
In NumPy:
p = np.linalg.lstsq(A, b)[0]p = np.linalg.svd(A)[2][-1]


Distortion acts on the image coordinates, before \(\mathbf{K}\): \[{}^{c}\mathbf{X} \;\to\; (x, y) = \Big(\tfrac{{}^cX}{{}^cZ}, \tfrac{{}^cY}{{}^cZ}\Big) \;\xrightarrow{\text{distortion}}\; (x_d, y_d) \;\xrightarrow{\;\mathbf{K}\;}\; (u, v)\]
With \(r^2 = x^2 + y^2\) (squared distance from the principal point): \[\begin{aligned} x_d &= \underbrace{\color{#2b6cb0}{x\,(1 + k_1 r^2 + k_2 r^4 + k_3 r^6)}}_{\color{#2b6cb0}{\text{radial}}} + \underbrace{\color{#c05621}{2 p_1 x y + p_2 (r^2 + 2 x^2)}}_{\color{#c05621}{\text{tangential}}} \\ y_d &= \underbrace{\color{#2b6cb0}{y\,(1 + k_1 r^2 + k_2 r^4 + k_3 r^6)}}_{\color{#2b6cb0}{\text{radial}}} + \underbrace{\color{#c05621}{p_1 (r^2 + 2 y^2) + 2 p_2 x y}}_{\color{#c05621}{\text{tangential}}} \end{aligned}\]
Pixels: \((u, v, 1)^{T} = \mathbf{K}\,(x_d, y_d, 1)^{T}\), instead of \(\mathbf{K}\,(x, y, 1)^{T}\). This is the camera model \(\pi(\cdot)\) of the previous slide.


cv2.fisheye: \(\theta_d = \theta\,(1 + k_1\theta^2 + k_2\theta^4 + k_3\theta^6 + k_4\theta^8)\)cv2.omnidir unified sphere model, or Scaramuzza’s polynomial model (OCamCalib).

| Target | OpenCV detector | Strengths | Weaknesses |
|---|---|---|---|
| Checkerboard | findChessboardCorners |
Simple; accurate corners (sub-pixel) | The whole board must be visible |
| Circle grids | findCirclesGrid |
Robust to blur and noise | Circle centers shift slightly under perspective |
| ChArUco | cv2.aruco.CharucoDetector |
Each corner is identified by its markers: partial and occluded views still work | Must be printed exactly to spec |

img = cv2.imread(str(calib_dir / "GOPR0046.jpg"))
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
pattern = (8, 6) # inner corners per row, per column
found, corners = cv2.findChessboardCorners(gray, pattern)
criteria = (cv2.TERM_CRITERIA_EPS
+ cv2.TERM_CRITERIA_MAX_ITER, 30, 1e-3)
corners_sub = cv2.cornerSubPix(gray, corners.copy(),
(11, 11), (-1, -1), criteria)corners: the 48 corner pixels, ordered row by row, starting at one end of the board.cornerSubPix refines each corner to sub-pixel accuracy using the image gradients in an \(11 \times 11\) window. It works in place, hence the .copy() here.found is False, skip the image: the board was not fully visible.
square = 80.0 # mm
objp = np.zeros((pattern[0] * pattern[1], 3), np.float32)
objp[:, :2] = np.mgrid[0:8, 0:6].T.reshape(-1, 2) * square
obj_points, img_points = [], []
for path in calib_files:
gray = cv2.imread(str(path), cv2.IMREAD_GRAYSCALE)
found, corners = cv2.findChessboardCorners(gray,
pattern)
if not found:
continue
corners = cv2.cornerSubPix(gray, corners, (11, 11),
(-1, -1), criteria)
obj_points.append(objp)
img_points.append(corners)objp holds the corners \({}^{b}\mathbf{X}_i = (X, Y, 0)\) in the board frame \(b\), in the same order as the detected corners.objp is the same for every view.
rms \(=\) 0.824 px: the RMS reprojection error over all 35 views.K:\[\mathbf{K} = \begin{bmatrix} 560 & 0 & 651.1 \\ 0 & 561.1 & 498.9 \\ 0 & 0 & 1 \end{bmatrix}\]
dist \(= (k_1, k_2, p_1, p_2, k_3)\):\[(-0.2326,\ 0.0615,\ -0.00003,\ 0.00006,\ -0.0075)\]
rvecs, tvecs: 35 board poses, one per view (next slide).| Output | Meaning |
|---|---|
K |
Intrinsic matrix \(\mathbf{K}\): \(f_x, f_y, c_x, c_y\) (skew fixed to 0) |
dist |
Distortion \(\mathbf{d} = (k_1, k_2, p_1, p_2, k_3)\) |
rvecs[j] |
Rotation \({}^{c}\mathbf{R}_{b_j}\) of view \(j\) (axis–angle) |
tvecs[j] |
Translation \({}^{c}\mathbf{t}_{b_j}\) of view \(j\) (mm) |
rms |
RMS reprojection error (px) |
cv2.calibrateCameraExtended returns parameter uncertainties and per-view errors in additional output arrays.

from pathlib import Path
import cv2
import numpy as np
# Calibration images and the checkerboard geometry.
calib_files = sorted(Path("assets/12_camera_calibration/calibration").glob("*.jpg"))
pattern = (8, 6) # inner corners per row, per column
square = 80.0 # square size in mm
# 3D corners in the board frame b: (X, Y, 0), in the same order as the detected corners.
objp = np.zeros((pattern[0] * pattern[1], 3), np.float32)
objp[:, :2] = np.mgrid[0:pattern[0], 0:pattern[1]].T.reshape(-1, 2) * square
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 1e-3)
# Steps 1-2: detect the corners in every image and match them to the board points.
obj_points, img_points = [], []
for path in calib_files:
gray = cv2.imread(str(path), cv2.IMREAD_GRAYSCALE)
found, corners = cv2.findChessboardCorners(gray, pattern)
if not found:
print(f"Board not found in {path.name}, skipping")
continue
corners = cv2.cornerSubPix(gray, corners, (11, 11), (-1, -1), criteria)
obj_points.append(objp)
img_points.append(corners)
image_size = gray.shape[::-1] # (width, height)
# Step 3: calibrate.
rms, K, dist, rvecs, tvecs = cv2.calibrateCamera(
obj_points, img_points, image_size, None, None)
print(f"Views used: {len(obj_points)} of {len(calib_files)}")
print(f"RMS reprojection error: {rms:.3f} px")
print("K =\n", K)
print("dist (k1, k2, p1, p2, k3) =", dist.ravel())
# Undistort an image with the calibrated K and dist.
img = cv2.imread(str(calib_files[0]))
undistorted = cv2.undistort(img, K, dist)
cv2.imwrite("undistorted.jpg", undistorted)

std_intrinsics: all well below one pixel.| All views | Outlier removed | |
|---|---|---|
| Views | 35 | 34 |
| RMS reprojection error (px) | 0.82 | 0.57 |
| \(f_x\) (px) | 560.0 \(\pm\) 1.0 | 560.4 \(\pm\) 0.7 |
| \(f_y\) (px) | 561.1 \(\pm\) 0.9 | 561.4 \(\pm\) 0.6 |
| \(c_x\) (px) | 651.1 \(\pm\) 0.3 | 651.1 \(\pm\) 0.2 |
| \(c_y\) (px) | 498.9 \(\pm\) 0.5 | 499.0 \(\pm\) 0.3 |
| \(k_1\) | -0.2326 \(\pm\) 0.0009 | -0.2328 \(\pm\) 0.0006 |
| \(k_2\) | 0.0615 \(\pm\) 0.0006 | 0.0616 \(\pm\) 0.0004 |
findChessboardCorners → cornerSubPix → calibrateCamera → undistort.Computer Vision; Colorado School of Mines | Kaveh Fathian