Minimize the sum of squared perpendicular distances while computing PCA

Question

Problem (PCA):

Assume that p = 2 and the the predictors are centered. Show that the sum of squared perpendicular distances from ($x_{i1}, x_{i2})$, i = 1, 2, . . . , n to the line $a_{2}x_{1}−a_{1}x_{2}=0$ with a $a_1^2 + a_2^2 = 1$ is minimized when $a = φ_{11}$ and $b = φ_{21}$, where $φ_1 = (φ_{11}, φ_{21})$ is a norm-one eigenvector associated with the eigenvalue $λ_1$.

I know that maximizing the variance along the principal component is equivalent to minimizing the reconstruction error,i.e., the sum of the squares of the perpendicular distance from the component. I don't know how can I integrate this idea here. I was wondering if you could give me some hints to solve this problem.

Answer 1

Sum of the squared distances from $x_i$' to the line is $$\sum_i(a_2x_{i1}-a_1x_{i2})^2=\sum_{i}(a_p^Tx_i)^2=\sum_i a_p^Tx_ix_i^Ta_p=a_p^T\left(\sum_i x_ix_i^T \right)a_p$$

The inside of the parentheses is the scatter matrix (i.e. scaled sampled covariance when predictors are centered, i.e. $X^TX$, call it as $S$). Also, $a_p=[a_2 \ -a_1]^T$ is the perpendicular vector to $a=[a_1\ a_2]^T$. The expression turns into quadratic form as below:

$$\min_{a_p \text{s.t.} ||a_p||=1} a_p^T Sa_p$$

This problem has solution where $a_p$ is the eigenvector corresponding to the minimum eigenvalue of $S$ (call it $\lambda_2$). Then, $a$ will be the eigenvector corresponding to the maximum eigenvalue, $\lambda_1$.

Note: Scatter matrix and Sample covariance matrix has only a scalar multiplication difference. So, the eigenvalues of these matrices will be only scaled versions of each other, and eigenvectors will be the same.