3
$\begingroup$

Problem (PCA):

Assume that p = 2 and the the predictors are centered. Show that the sum of squared perpendicular distances from ($x_{i1}, x_{i2})$, i = 1, 2, . . . , n to the line $a_{2}x_{1}−a_{1}x_{2}=0$ with a $a_1^2 + a_2^2 = 1$ is minimized when $a = φ_{11}$ and $b = φ_{21}$, where $φ_1 = (φ_{11}, φ_{21})$ is a norm-one eigenvector associated with the eigenvalue $λ_1$.

I know that maximizing the variance along the principal component is equivalent to minimizing the reconstruction error,i.e., the sum of the squares of the perpendicular distance from the component. I don't know how can I integrate this idea here. I was wondering if you could give me some hints to solve this problem.

$\endgroup$
16
  • $\begingroup$ Here, $\lambda_1$ should be the minimum eigenvalue of the data covariance matrix. Can you please verify it? Because, the sum of squared errors is minimised along the first eigenvector's direction, but that line is not represented by the first eigenvector but the second. A line $ax+by=0$ has the vector $[a,b]$ as its normal (it's not parallel). $\endgroup$
    – gunes
    Dec 5 '20 at 14:38
  • $\begingroup$ @gunes How can I verify this? $\endgroup$ Dec 5 '20 at 14:59
  • $\begingroup$ Checking the source of the question maybe? Is it a book? The question doesn’t specify exactly what $\lambda_1$ mean. $\endgroup$
    – gunes
    Dec 5 '20 at 15:06
  • 1
    $\begingroup$ By using the information in my post at stats.stackexchange.com/a/71303/919 you should have no trouble obtaining this result. $\endgroup$
    – whuber
    Dec 5 '20 at 15:16
  • 1
    $\begingroup$ @gunes I have got some corrections. The line equation is $a_{2}x_1 − a_{1}x_2 = 0$ $\endgroup$ Dec 7 '20 at 1:20
1
$\begingroup$

Sum of the squared distances from $x_i$' to the line is $$\sum_i(a_2x_{i1}-a_1x_{i2})^2=\sum_{i}(a_p^Tx_i)^2=\sum_i a_p^Tx_ix_i^Ta_p=a_p^T\left(\sum_i x_ix_i^T \right)a_p$$

The inside of the parentheses is the scatter matrix (i.e. scaled sampled covariance when predictors are centered, i.e. $X^TX$, call it as $S$). Also, $a_p=[a_2 \ -a_1]^T$ is the perpendicular vector to $a=[a_1\ a_2]^T$. The expression turns into quadratic form as below:

$$\min_{a_p \text{s.t.} ||a_p||=1} a_p^T Sa_p$$

This problem has solution where $a_p$ is the eigenvector corresponding to the minimum eigenvalue of $S$ (call it $\lambda_2$). Then, $a$ will be the eigenvector corresponding to the maximum eigenvalue, $\lambda_1$.

Note: Scatter matrix and Sample covariance matrix has only a scalar multiplication difference. So, the eigenvalues of these matrices will be only scaled versions of each other, and eigenvectors will be the same.

$\endgroup$
3
  • $\begingroup$ Thank you so much! One more thing, Is it related to total least squares? I would like to learn more about this. $\endgroup$ Dec 7 '20 at 14:24
  • $\begingroup$ Although I don't know the explicit connection, they should be. $\endgroup$
    – gunes
    Dec 7 '20 at 14:33
  • $\begingroup$ It's okay! I really appreciate your effort! Stay safe! $\endgroup$ Dec 7 '20 at 14:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.