I have a dataset of 64 highly-correlated variables (with many being interactions of others for example) plus a date and the target variable for a total of 66. I want to derive the principal components of the 64 independent variables and choose the final predictors among those (I don't want to use principal components regression directly as it must choose the first n components instead of choosing the statistically significant ones).
Code in R:
PC=prcomp(mydata[,3:66], center=TRUE, scale=TRUE) PrincipalComponents=as.data.frame(PC$x)
I would expect Principal Components to be orthogonal and essentially uncorrelated (as apparently these aren't quite as interchangeable as I had thought).
returns a number very close to 0, as expected and so do most pair-wise correlations. However,
cor(PrincipalComponents$PC1, PrincipalComponents$PC63) cor(PrincipalComponents$PC1, PrincipalComponents$PC64)
return 0.61 and 0.41. These high correlations are only observed for the last two principal components. Running a regression using the principal components as independent variables, I get a VIF>10 on nearly 30 of them. Running a regression but excluding the last two principal components, I get VIF=1 for all predictors.
Am I missing something? If that is something that is expected, doesn't that defeat the purpose of PCA? I'd like to understand what is going on here before I make the decision to exclude those 2 components for example.