PCA leads to some highly Correlated Principal Components

Question

I have a dataset of 64 highly-correlated variables (with many being interactions of others for example) plus a date and the target variable for a total of 66. I want to derive the principal components of the 64 independent variables and choose the final predictors among those (I don't want to use principal components regression directly as it must choose the first n components instead of choosing the statistically significant ones).

Code in R:

PC=prcomp(mydata[,3:66], center=TRUE, scale=TRUE)
PrincipalComponents=as.data.frame(PC$x)

I would expect Principal Components to be orthogonal and essentially uncorrelated (as apparently these aren't quite as interchangeable as I had thought).

cor(PrincipalComponents$PC1, PrincipalComponents$PC2)

returns a number very close to 0, as expected and so do most pair-wise correlations. However,

cor(PrincipalComponents$PC1, PrincipalComponents$PC63)
cor(PrincipalComponents$PC1, PrincipalComponents$PC64)

return 0.61 and 0.41. These high correlations are only observed for the last two principal components. Running a regression using the principal components as independent variables, I get a VIF>10 on nearly 30 of them. Running a regression but excluding the last two principal components, I get VIF=1 for all predictors.

Am I missing something? If that is something that is expected, doesn't that defeat the purpose of PCA? I'd like to understand what is going on here before I make the decision to exclude those 2 components for example.

Answer 1

You're right that the principal components should all be mutually orthogonal, so this is not expected. I think you probably have columns in your data matrix which are linearly dependent.

If the column rank of your data matrix is < 64, it is not possible to find 64 mutually orthogonal vectors in its column space. It might be better behavior for the package to return an error, or maybe fewer than 64 columns, in this case, but that is not what prcomp does.

Example:

m <- matrix(c(1,0,0,0,0,0,1,0,0), nrow=3)
cor(prcomp(m)$x)

Claims there are three principal components with pairwise correlation +/- 1. Something similar is probably happening in your data.