I've seen some great posts explaining PCA and why under this approach the eigenvectors of a (symmetric) correlation matrix are orthogonal. I also understand the ways to show that such vectors are orthogonal to each other (e.g. taking the cross-products of the matrix of these eigenvectors will result in a matrix with off-diagonal entries that are zero).

My first question is, when you look at the correlations of a PCA's eigenvectors, why are the off-diagonal entries of the correlation matrix non-zero (i.e. how can the eigenvectors be correlated if they are orthogonal)?

This question is not directly about PCA, but I put it in this context since that is how I ran into the issue. I am using R and specifically the psych package to run PCA.

If it helps to have an example, this post on StackOverflow has one that is very convenient and related (also in R). In this post, the author of the best answer shows that the PCA loadings (eigenvectors) are orthogonal by using Factor Congruence or cross-products. In his example, the matrix L is the PCA loadings matrix. The only thing that is not on this link is that cor(L) will produce the output I am asking about showing the non-zero correlations between the eigenvectors.

I am especially confused about how orthogonal vectors can be correlated after reading this post, which seems to prove that orthogonality is equivalent to lack of correlation: Why are PCA eigenvectors orthogonal and what is the relation to the PCA scores being uncorrelated?

My second question is: when the PCA eigenvectors are used to calculate PCA scores, the scores themselves are uncorrelated (as I expected)... is there a connection to my first question about this, why eigenvectors are correlated but not the scores?

  • 1
    $\begingroup$ It sounds like you're computing the correlation matrix of the eigenvectors. The eigenvectors are orthogonal, implying the dot products between them are zero, not the correlations. What should be uncorrelated is the projections of the data onto the eigenvectors, not the eigenvectors themselves. $\endgroup$ – user20160 Jan 24 '18 at 6:24
  • 1
    $\begingroup$ Eigenvectors are just fixed quantities, not random variables. It doesn't make sense to talk about correlations between them because they don't fluctuate. $\endgroup$ – Moss Murderer Jan 24 '18 at 6:48
  • 2
    $\begingroup$ The key is that eigenvectors do not have mean zero (e.g. it's completely possible that all elements of an eigenvector are positive). So the fact that they are orthogonal does not imply that their elements are uncorrelated (to compute correlation we need to subtract the means). $\endgroup$ – amoeba Jan 24 '18 at 8:25
  • 2
    $\begingroup$ @Moss On the contrary, the eigenvectors output by PCA, since they are functions of the data, are indeed random variables. That might be what is leading to confusion because now "orthogonality... of eigenvectors" has multiple interpretations: orthogonality as Euclidean vectors, lack of correlation of their components (as random variables), and even orthogonality of the eigenvectors of the true underlying multivariate distribution presumed to generate the data. $\endgroup$ – whuber Jan 24 '18 at 14:52
  • $\begingroup$ @whuber: That makes sense. I was thinking of eigenvectors as a parameter rather than as a statistic. $\endgroup$ – Moss Murderer Jan 24 '18 at 15:07

Let $X$ be a random vector $X=(x_1,x_2,\cdots,x_d)^T $ with expected value $\mu$ and variance $\Sigma$. We are looking for such ordered vectors $u_i$, that maximize the variance of $u_i^TX$. Essentialy we are solving $$\max\limits_{u_i} Var(u_i^TX)$$ $$s.t. \quad u_i^T u_i=1.$$ Because we are only interested in the direction of such vectors, we are additionally assuming the unit length of vectors $u_i^T u_i=1$. Vectors $u_i$ are actually not random (because we are working theoretically now, in reality we are replacing the unknown $\Sigma$ and unknown $\mu$ with Empirical sample covariance matrix and mean respectively, @whuber was explaining this from a different perspective) so $$Var(u_i^TX)=u_i^T\Sigma u_i.$$ The optimization problem can be trivially solved by using the Lagrange function $$L(u_i,\lambda_i):=u_i^T \Sigma u_i -\lambda_i(u_i^Tu_i-1).$$ From there we get the necessary condition for constrained extrema $$ \frac{\partial L(u_i,\lambda_i)}{\partial u_i} = 2\Sigma u_i -2\lambda_i u_i=0,$$ which can be reduced to $$\Sigma u_i =\lambda_i u_i,$$ that is by definition the problem of eigenvalues and eigenvectors. Because $\Sigma$ is symmetric and positive semidefinite matrix, the spectral theorem applies and we are able to find orthonormal basis that satisfies $\Sigma=Q\Lambda Q^{-1}=Q\Lambda Q^T$, where $Q$ is made of orthogonal eigenvectors and $\Lambda$ is a diagonal matrix with eigenvalues which are all real.

Now we can show that $$cov(u_i^TX,u_j^TX)=u_i^T\Sigma u_j=\lambda_j u_i^Tu_j=0, \quad \forall j \neq i.$$ Trivially for $i=j: \quad cov(u_i^TX,u_j^TX)=\lambda_i.$ So not the eigenvectors, but the projections are uncorrelated.

| cite | improve this answer | |

Note that L is the loadings matrix, aka the eigenvectors themselves. This isn't the PCA data matrix. The eigenvectors are bound to provide orthogonality but not $cov=0$. For example, take the matrix:

> X <- iris
> X$Species <- as.numeric(X$Species)
> head(X)
  Sepal.Length Sepal.Width Petal.Length Petal.Width Species
1          5.1         3.5          1.4         0.2       1
2          4.9         3.0          1.4         0.2       1
3          4.7         3.2          1.3         0.2       1
4          4.6         3.1          1.5         0.2       1
5          5.0         3.6          1.4         0.2       1
6          5.4         3.9          1.7         0.4       1

In PCA, not only you get the eigenvectors of the covariance/correlation matrix (depends on the method) but they are also being orthonormal (that is, $\left \| u_j \right \|=1$ for each eigenvector $u_j$), so we get:

> prcomp(X)$rotation
                     PC1         PC2        PC3         PC4        PC5
Sepal.Length  0.33402494 -0.68852577  0.4414776 -0.43312829  0.1784853
Sepal.Width  -0.08034626 -0.68474905 -0.6114140  0.30348725 -0.2423462
Petal.Length  0.80059273  0.09713877  0.1466787  0.49080356 -0.2953177
Petal.Width   0.33657862  0.06894557 -0.4202025  0.06667133  0.8372253
Species       0.35740442  0.20703034 -0.4828930 -0.68917499 -0.3482135


> cor(prcomp(X)$rotation)
            PC1         PC2         PC3          PC4          PC5
PC1  1.00000000  0.62712979  0.57079328  0.147574029 -0.072934736
PC2  0.62712979  1.00000000 -0.22763304 -0.058852698  0.029086459
PC3  0.57079328 -0.22763304  1.00000000 -0.053565825  0.026473556
PC4  0.14757403 -0.05885270 -0.05356582  1.000000000  0.006844526
PC5 -0.07293474  0.02908646  0.02647356  0.006844526  1.000000000

but note that the PCA'd data is

> head(prcomp(X)$x)
           PC1        PC2          PC3          PC4          PC5
[1,] -2.865415 -0.2962946 -0.041870662 -0.078464301 -0.032047052
[2,] -2.892047  0.1837851  0.175540800 -0.143582265  0.053428970
[3,] -3.054980  0.1748266 -0.049705391 -0.045339514 -0.001205543
[4,] -2.920230  0.3315818 -0.003376012  0.065785303 -0.053882996
[5,] -2.906852 -0.2959169 -0.147159821 -0.004802747 -0.074130194
[6,] -2.489852 -0.7338212 -0.194029844  0.073567444  0.003409809

and its correlation is

> round(cor(prcomp(X)$x),14)
    PC1 PC2 PC3 PC4 PC5
PC1   1   0   0   0   0
PC2   0   1   0   0   0
PC3   0   0   1   0   0
PC4   0   0   0   1   0
PC5   0   0   0   0   1
| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.