Why are PCA eigenvectors orthogonal and what is the relation to the PCA scores being uncorrelated?

Question

I'm reading up on PCA, and I'm understanding most of what's going on in terms of the derivation apart from the assumption that eigenvectors need to be orthogonal and how it relates to the projections (PCA scores) being uncorrelated? I have two explanations provided below which use a link between orthogonality and correlation but fail to really explain it: ONE, TWO.

In the second picture it says that the condition $a_{2}^{T}a_{1}=0$ is imposed to ensure that the projection $y_{2}=Xa_2$ will be uncorrelated with $y_{1}=Xa_1$. Can someone provide an example to show why orthogonal vectors ensure uncorrelated variables?

What would happen in PCA if I chose vectors which are not orthogonal; is this even possible? I have read elsewhere that orthogonality is just a by-product of the covariance matrix being symmetric which would suggest that it's not possible to have non pairwise orthogonal eigenvectors. However in the first picture in search of the most 'suitable' matrix it almost seems as we are choosing $p_{1},\ldots,p_{m}$ to be orthogonal to give us a more convenient matrix $\textbf{P}$ one which has nice properties.

I've read other posts on this topic but have been unsatisfied with incorporation of the intuition with uncorrelated variables. I really appreciate any help in understanding this confusion!!

Answer 1

I will try to explain how orthogonality of $a_1$ and $a_2$ ensures that $y_1$ and $y_2$ be uncorrelated. We want $a_1$ to maximize $Var(y_1)=a_1^T \Sigma a_1$. This will not be achieved unless we constrain $a_1$, in this case by $a_1^T a_1=1$. This optimization calls for the use of a Lagrange Multiplier (it's not too complicated, read about it on Wikipedia). We thus try to maximize \begin{equation} a_1^T \Sigma a_1 - \lambda(a_1^T a_1-1) \end{equation} with respect to both $a_1$ and $\lambda$. Notice that differentiation with respect to $\lambda$ and then equating to $0$ gives our constraint $a_1^T a_1=1$. Differentiation with respect to $a_1$ gives \begin{equation} \Sigma a_1 -\lambda a_1 =0 \end{equation} or \begin{equation} (\Sigma -\lambda I_p)a_1=0 \end{equation} variance of $y_1$ will be maximized by the greatest eigenvalue $\lambda_1$. Thus $\lambda_1 a_1=\Sigma a_1$. Here comes the part that will answer your question. Some elementary calculations using the definition of covariance will show that \begin{equation} Cov(y_1,y_2)=Cov(a^T_1 x,a^T_2 x)=a^T_1\Sigma a_2=a^T_2\Sigma a_1=a^T_2\lambda_1 a_1=\lambda_1 a^T_2 a_1 \end{equation} which will equal $0$ if and only if $a^T_2 a_1=0$.

Answer 2

PCA works by computing the eigenvectors of the covariance matrix of the data. That is, those eigenvectors correspond to the choices of $a_{1:M}$ that maximize the equations and meet the constraints given in your book. If you chose different vectors, they wouldn't fit all those criteria, and it wouldn't be a PCA anymore (you would still find a number of "components" but they would no longer be "principal").

Eigenvectors can be computed from any square matrix and don't have to be orthogonal. However, since any proper covariance matrix is symmetric, and symmetric matrices have orthogonal eigenvectors, PCA always leads to orthogonal components.

The orthogonality of $y_1$ and $y_2$ doesn't follow solely from the requirement that $a_1^Ta_2=0$ - it follows from all the constraints together. It's easy to see why orthogonality of $a_1$ and $a_2$ isn't enough, because the original basis $\mathbf{b}$ in which the data is expressed is also orthogonal. E.g. in 2 dimensions, you would have $b_1=\begin{bmatrix}1\\ 0 \end{bmatrix}$ and $b_2=\begin{bmatrix}0\\ 1 \end{bmatrix}$ and clearly your data don't have to be uncorrelated along those dimensions (if they were, your PCA would just return the original basis, up to a scaling factor).

The text is worded a bit awkwardly, but I think the "which" in "which ensures..." refers to the entire clause that came before.