
Lecture 11
Computer Science Department, Colorado School of Mines
By the end of this lecture, you should be able to:

Frisius’s woodcut is the oldest known published drawing of a camera obscura.




Placing a virtual image plane a distance \(f\) in front of the camera center gives the same image, the same size, but upright.
The virtual plane is mathematically equivalent and easier to work with, so from now on we will use it instead.


Side view (\(y\)–\(z\) plane): the two shaded triangles are similar, so \[\frac{y}{f} = \frac{Y}{Z}, \qquad \text{and likewise} \qquad \frac{x}{f} = \frac{X}{Z}.\]
Perspective projection: \[(X,\ Y,\ Z) \;\mapsto\; \left(f\,\frac{X}{Z},\; f\,\frac{Y}{Z}\right)\]
The division by \(Z\) makes the projection nonlinear in \((X, Y, Z)\).
Homogeneous coordinates (defined later) will make it a matrix product.


StruckDuck, Impossible Gravity Slide Illusion
| 1. Image Filtering | \(\displaystyle y[m,n]=\sum_{k,l} h[k,l]\,I[m+k,n+l]\) |
| 2. Optical Flow | \(\displaystyle I(x,y,t)=I(x+\Delta x,\ y+\Delta y,\ t+\Delta t)\) |
| 3. Camera Geometry | \(\displaystyle \lambda\,\mathbf{x}=\mathbf{K}\begin{bmatrix}{}^{c}\mathbf{R}_{w}&{}^{c}\mathbf{t}_{w}\end{bmatrix}\,{}^{w}\mathbf{X} \qquad \mathbf{x}^{T}\mathbf{F}\mathbf{x}'=0\) |
| 4. Machine Learning | \(\displaystyle \underset{\mathcal{S}}{\operatorname{arg\,min}}\sum_{i=1}^{k}\sum_{\mathbf{x}\in S_i}\left\lVert\mathbf{x}-\boldsymbol{\mu}_i\right\rVert^{2} \qquad y=\varphi\!\left(\mathbf{w}^{T}\mathbf{x}+b\right)\) |

| To homogeneous: append a 1 | From homogeneous: divide by the last coordinate | |
|---|---|---|
| 2D (image) | \((x, y) \;\Rightarrow\; \begin{bmatrix} x \\ y \\ 1 \end{bmatrix}\) | \(\begin{bmatrix} x \\ y \\ w \end{bmatrix} \;\Rightarrow\; \left(\dfrac{x}{w},\ \dfrac{y}{w}\right)\) |
| 3D (scene) | \((x, y, z) \;\Rightarrow\; \begin{bmatrix} x \\ y \\ z \\ 1 \end{bmatrix}\) | \(\begin{bmatrix} x \\ y \\ z \\ w \end{bmatrix} \;\Rightarrow\; \left(\dfrac{x}{w},\ \dfrac{y}{w},\ \dfrac{z}{w}\right)\) |
\[ \begin{bmatrix} p'_x \\ p'_y \end{bmatrix} = \underbrace{\begin{bmatrix} m_{11} & m_{12} \\ m_{21} & m_{22} \end{bmatrix}}_{\mathbf{M}} \begin{bmatrix} p_x \\ p_y \end{bmatrix} . \]



| Translation | Uniform scale | Non-uniform scale | Shear | Rotation | |
|---|---|---|---|---|---|
| Cartesian | none: \(\mathbf{p}' = \mathbf{p} + \mathbf{t}\) | \(\begin{bmatrix} s & 0 \\ 0 & s \end{bmatrix}\) | \(\begin{bmatrix} s_x & 0 \\ 0 & s_y \end{bmatrix}\) | \(\begin{bmatrix} 1 & a_x \\ a_y & 1 \end{bmatrix}\) | \(\begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix}\) |
| Homogeneous | \(\begin{bmatrix} 1 & 0 & t_x \\ 0 & 1 & t_y \\ 0 & 0 & 1 \end{bmatrix}\) | \(\begin{bmatrix} s & 0 & 0 \\ 0 & s & 0 \\ 0 & 0 & 1 \end{bmatrix}\) | \(\begin{bmatrix} s_x & 0 & 0 \\ 0 & s_y & 0 \\ 0 & 0 & 1 \end{bmatrix}\) | \(\begin{bmatrix} 1 & a_x & 0 \\ a_y & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}\) | \(\begin{bmatrix} \cos\theta & -\sin\theta & 0 \\ \sin\theta & \cos\theta & 0 \\ 0 & 0 & 1 \end{bmatrix}\) |
\[\mathbf{R} = \begin{bmatrix} \cos 45^\circ & -\sin 45^\circ & 0 \\ \sin 45^\circ & \cos 45^\circ & 0 \\ 0 & 0 & 1 \end{bmatrix}\qquad\qquad \mathbf{T} = \begin{bmatrix} 1 & 0 & 2.2 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}\]

\[\mathbf{T}\,\mathbf{R} = \begin{bmatrix} 0.71 & -0.71 & 2.2 \\ 0.71 & 0.71 & 0 \\ 0 & 0 & 1 \end{bmatrix}\]

\[\mathbf{R}\,\mathbf{T} = \begin{bmatrix} 0.71 & -0.71 & 1.56 \\ 0.71 & 0.71 & 1.56 \\ 0 & 0 & 1 \end{bmatrix}\]

| Transformation | Matrix | DOF | Preserves (and everything below) |
|---|---|---|---|
| Translation | \(\begin{bmatrix} \mathbf{I} & \mathbf{t} \\ \mathbf{0}^{T} & 1 \end{bmatrix}\) | 2 | orientation |
| Rigid (Euclidean) | \(\begin{bmatrix} \mathbf{R} & \mathbf{t} \\ \mathbf{0}^{T} & 1 \end{bmatrix}\) | 3 | lengths |
| Similarity | \(\begin{bmatrix} s\mathbf{R} & \mathbf{t} \\ \mathbf{0}^{T} & 1 \end{bmatrix}\) | 4 | angles |
| Affine | \(\begin{bmatrix} \mathbf{A} & \mathbf{t} \\ \mathbf{0}^{T} & 1 \end{bmatrix}\) | 6 | parallelism |
| Projective | \(\mathbf{H}\) (any invertible \(3\times3\)) | 8 | straight lines |


| Translation | Rotation | Rotation + translation (rigid) | |
|---|---|---|---|
| Cartesian | none: \(\mathbf{X}' = \mathbf{X} + \mathbf{t}\) | \(\mathbf{X}' = \mathbf{R}\,\mathbf{X}\) | \(\mathbf{X}' = \mathbf{R}\,\mathbf{X} + \mathbf{t}\) |
| Homogeneous (\(4\times4\)) | \(\begin{bmatrix} \mathbf{I} & \mathbf{t} \\ \mathbf{0}^{T} & 1 \end{bmatrix}\) | \(\begin{bmatrix} \mathbf{R} & \mathbf{0} \\ \mathbf{0}^{T} & 1 \end{bmatrix}\) | \(\begin{bmatrix} \mathbf{R} & \mathbf{t} \\ \mathbf{0}^{T} & 1 \end{bmatrix}\) |


| 1. Extrinsics | 2. Perspective projection | 3. Intrinsics |
|---|---|---|
| world frame \(\to\) camera frame | camera frame \(\to\) image plane | image plane \(\to\) pixels |
Each step is a matrix in homogeneous coordinates, so the whole camera model is one matrix.


\[\underbrace{\begin{bmatrix} \tfrac{1}{\rho_x} & 0 & c_x \\ 0 & \tfrac{1}{\rho_y} & c_y \\ 0 & 0 & 1 \end{bmatrix}}_{\text{to pixels}} \underbrace{\begin{bmatrix} f & 0 & 0 \\ 0 & f & 0 \\ 0 & 0 & 1 \end{bmatrix}}_{\text{focal length}} = \begin{bmatrix} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix}\]
\[\mathbf{K} = \begin{bmatrix} f_x & s & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix}\]
Chaining the three steps, with \(\mathbf{x} = (u, v, 1)\) in pixels and \(\lambda = Z_c\): \[ \begin{aligned} \lambda\,\mathbf{x} &= \underbrace{\mathbf{K}}_{3\times3}\; \underbrace{\begin{bmatrix} \mathbf{I} & \mathbf{0} \end{bmatrix}}_{3\times4}\; \underbrace{{}^{c}\mathbf{T}_{w}}_{4\times4}\; \underbrace{{}^{w}\mathbf{X}}_{4\times1} \\[4pt] &= \underbrace{\mathbf{K}}_{\substack{\text{intrinsic}\\ \text{matrix}}}\; \underbrace{\big[\,{}^{c}\mathbf{R}_{w} \mid {}^{c}\mathbf{t}_{w}\,\big]}_{\substack{\text{extrinsic}\\ \text{matrix}}}\;{}^{w}\mathbf{X} = \mathbf{P}\;{}^{w}\mathbf{X} \end{aligned} \]
| Part | Parameters | DOF |
|---|---|---|
| Intrinsic matrix \(\mathbf{K}\) | \(f_x,\ f_y,\ c_x,\ c_y,\ s\) | 5 |
| Extrinsic matrix \(\big[\,{}^{c}\mathbf{R}_{w} \mid {}^{c}\mathbf{t}_{w}\,\big]\) | rotation (3) + translation (3) | 6 |
| Camera matrix \(\mathbf{P}\) | 11 |
For the camera and the red corner in the cube example:
\[\underbrace{\begin{bmatrix} 500 & 0 & 320 \\ 0 & 500 & 240 \\ 0 & 0 & 1 \end{bmatrix}}_{\mathbf{K}:\ \text{intrinsic matrix}}\qquad\qquad\underbrace{\begin{bmatrix} 0.77 & 0.63 & 0 & -1.27 \\ 0.23 & -0.29 & -0.93 & 0.51 \\ -0.59 & 0.72 & -0.37 & 4.66 \end{bmatrix}}_{\big[\,{}^{c}\mathbf{R}_{w} \mid {}^{c}\mathbf{t}_{w}\,\big]:\ \text{extrinsic matrix}}\]
\[\mathbf{P} = \mathbf{K}\,\big[\,{}^{c}\mathbf{R}_{w} \mid {}^{c}\mathbf{t}_{w}\,\big] = \begin{bmatrix} 198.75 & 546.67 & -118.51 & 857.24 \\ -23.91 & 29.22 & -553.33 & 1373.53 \\ -0.59 & 0.72 & -0.37 & 4.66 \end{bmatrix}\]
\[\lambda\,\mathbf{x} = \mathbf{P}\;{}^{w}\mathbf{X} = \mathbf{P}\,({1.4},\ {1.4},\ {1.0},\ {1.0})^{T} = ({1782.3},\ {827.6},\ {4.5})\;\Rightarrow\; (u, v) = ({398.7},\ {185.1})\]
K = np.array([[500.0, 0.0, 320.0], [0.0, 500.0, 240.0], [0.0, 0.0, 1.0]]) # intrinsics
rvec = np.array([0.3, -0.4, 0.1])
R_c_w, _ = cv2.Rodrigues(rvec) # extrinsics rotation
t_c_w = np.array([[-0.4], [-0.3], [3.0]]) # extrinsics translation
P = K @ np.hstack([R_c_w, t_c_w]) # camera matrix
X_w = np.array([[x, y, z] for x in (0, 1) for y in (0, 1) for z in (0, 1)], dtype=float) # cube corners
X_w_h = np.hstack([X_w, np.ones((len(X_w), 1))]) # homogeneous coordinates
x = (P @ X_w_h.T).T # projection (lambda * u, lambda * v, lambda)
uv = x[:, :2] / x[:, 2:] # pixel coordinates (u, v)rvec: the rotation as a 3-vector (axis \(\times\) angle); cv2.Rodrigues turns it into \({}^{c}\mathbf{R}_{w}\).P \(= \mathbf{K}\,[\,{}^{c}\mathbf{R}_{w} \mid {}^{c}\mathbf{t}_{w}\,]\).X_w_h holds the corners as homogeneous rows.x is \(\lambda\,(u, v, 1)\) with \(\lambda = Z_c\); dividing by it gives uv.
projectPoints takes the rotation as rvec and applies \(\mathbf{K}\,[\,{}^{c}\mathbf{R}_{w} \mid {}^{c}\mathbf{t}_{w}\,]\) internally.reshape(-1, 2) gives \(N \times 2\) pixel coordinates.None here.

\[ \underbrace{\lambda}_{\text{depth } Z_c}\, \underbrace{\mathbf{x}}_{\substack{\text{pixel}\\ (u,\,v,\,1)}} = \overbrace{ \underbrace{\mathbf{K}}_{\substack{\text{intrinsics}\\ 3\times3}}\; \underbrace{\big[\,{}^{c}\mathbf{R}_{w} \mid {}^{c}\mathbf{t}_{w}\,\big]}_{\substack{\text{extrinsics}\\ 3\times4}} }^{\mathbf{P}:\ \text{camera matrix},\ 3\times4}\; \underbrace{{}^{w}\mathbf{X}}_{\substack{\text{world point}\\ (X,\,Y,\,Z,\,1)}} \]
Divide by the last coordinate to recover the pixel coordinates \((u,v)\).
Computer Vision; Colorado School of Mines | Kaveh Fathian