I'm reading up on PCA, and I'm understanding most of what's going on in terms of the derivation apart from the assumption that eigenvectors need to be orthogonal and how it relates to the projections (PCA scores) being uncorrelated? I have two explanations provided below which use a link between orthogonality and correlation but fail to really explain it: ONE, TWO.

In the second picture it says that the condition $a_{2}^{T}a_{1}=0$ is imposed to ensure that the projection $y_{2}=Xa_2$ will be uncorrelated with $y_{1}=Xa_1$. Can someone provide an example to show why orthogonal vectors ensure uncorrelated variables?

What would happen in PCA if I chose vectors which are not orthogonal; is this even possible? I have read elsewhere that orthogonality is just a by-product of the covariance matrix being symmetric which would suggest that it's not possible to have non pairwise orthogonal eigenvectors. However in the first picture in search of the most 'suitable' matrix it almost seems as we are choosing $p_{1},\ldots,p_{m}$ to be orthogonal to give us a more convenient matrix $\textbf{P}$ one which has nice properties.

I've read other posts on this topic but have been unsatisfied with incorporation of the intuition with uncorrelated variables. I really appreciate any help in understanding this confusion!!

  • $\begingroup$ The dot product of centered vectors is always proportional to their covariance, which in turn is proportional to their correlation. This is immediate: the formulas for all three are the same up to a nonzero constant. Thus one is zero if and only if the any other one is zero. $\endgroup$ – whuber Mar 10 '17 at 14:38
  • $\begingroup$ @whuber I think you misunderstood the question: OP is asking how orthogonality of the PCA eigenvectors implies zero correlation of the data projections onto these eigenvectors. $\endgroup$ – amoeba Mar 10 '17 at 14:46
  • $\begingroup$ @Amoeba I'm afraid that mystifies me even more. If vectors are orthogonal, then a fortiori any projections on those vectors must be orthogonal. The question I was responding to is "can someone provide an example to show why orthogonal vectors ensure uncorrelated variables." That still seems to ask why orthogonality implies lack of correlation. $\endgroup$ – whuber Mar 10 '17 at 14:52
  • $\begingroup$ @whuber What do you mean "If vectors are orthogonal, then a fortiori any projections on those vectors must be orthogonal"? There is some misunderstanding here. Take any bivariate data with non-zero correlation. Vectors [0,1] and [1,0] (basis vectors) are orthogonal, but data projections onto these vectors are correlated. $\endgroup$ – amoeba Mar 10 '17 at 14:53
  • $\begingroup$ @Amoeba I'm sure you're correct and I'm equally sure we have two different understandings of what you're saying! If you project a vector $p_1$ onto a vector $v$ and a vector $p_2$ onto $w$ and $v$ and $w$ are orthogonal, then the projections will be orthogonal too. If instead you project the $p_i$ onto the vector space generated by $\{v,w\}$ then of course the projections need not be orthogonal. These trivialities aren't worth discussing: our first concern should be to clarify what the question actually is asking for. $\endgroup$ – whuber Mar 10 '17 at 14:56

I will try to explain how orthogonality of $a_1$ and $a_2$ ensures that $y_1$ and $y_2$ be uncorrelated. We want $a_1$ to maximize $Var(y_1)=a_1^T \Sigma a_1$. This will not be achieved unless we constrain $a_1$, in this case by $a_1^T a_1=1$. This optimization calls for the use of a Lagrange Multiplier (it's not too complicated, read about it on Wikipedia). We thus try to maximize \begin{equation} a_1^T \Sigma a_1 - \lambda(a_1^T a_1-1) \end{equation} with respect to both $a_1$ and $\lambda$. Notice that differentiation with respect to $\lambda$ and then equating to $0$ gives our constraint $a_1^T a_1=1$. Differentiation with respect to $a_1$ gives \begin{equation} \Sigma a_1 -\lambda a_1 =0 \end{equation} or \begin{equation} (\Sigma -\lambda I_p)a_1=0 \end{equation} variance of $y_1$ will be maximized by the greatest eigenvalue $\lambda_1$. Thus $\lambda_1 a_1=\Sigma a_1$. Here comes the part that will answer your question. Some elementary calculations using the definition of covariance will show that \begin{equation} Cov(y_1,y_2)=Cov(a^T_1 x,a^T_2 x)=a^T_1\Sigma a_2=a^T_2\Sigma a_1=a^T_2\lambda_1 a_1=\lambda_1 a^T_2 a_1 \end{equation} which will equal $0$ if and only if $a^T_2 a_1=0$.


PCA works by computing the eigenvectors of the covariance matrix of the data. That is, those eigenvectors correspond to the choices of $a_{1:M}$ that maximize the equations and meet the constraints given in your book. If you chose different vectors, they wouldn't fit all those criteria, and it wouldn't be a PCA anymore (you would still find a number of "components" but they would no longer be "principal").

Eigenvectors can be computed from any square matrix and don't have to be orthogonal. However, since any proper covariance matrix is symmetric, and symmetric matrices have orthogonal eigenvectors, PCA always leads to orthogonal components.

The orthogonality of $y_1$ and $y_2$ doesn't follow solely from the requirement that $a_1^Ta_2=0$ - it follows from all the constraints together. It's easy to see why orthogonality of $a_1$ and $a_2$ isn't enough, because the original basis $\mathbf{b}$ in which the data is expressed is also orthogonal. E.g. in 2 dimensions, you would have $b_1=\begin{bmatrix}1\\ 0 \end{bmatrix}$ and $b_2=\begin{bmatrix}0\\ 1 \end{bmatrix}$ and clearly your data don't have to be uncorrelated along those dimensions (if they were, your PCA would just return the original basis, up to a scaling factor).

The text is worded a bit awkwardly, but I think the "which" in "which ensures..." refers to the entire clause that came before.

  • $\begingroup$ Thanks for the great post, i think for me to full understand it could you elaborate on two points, 1. How the conditions ensure the orthogonality of $y_{1}$ and $y_{2}$ and 2. how orthogonality then becomes leads to the variables being uncorrelated? Maybe by means of a proof or an example? $\endgroup$ – Pavan Sangha Mar 13 '17 at 21:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.