Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 2 down vote favorite

I'm looking for clarification: Do we have to run correlation analysis before carrying out principal component analysis (PCA), or is this implicitly subsumed under PCA framework?

I am on my way to write a research plan:

  • Isolate predictors with high collinear relationship to the process.
  • Run PCA to convert correlated predictors to uncorrelated principal component variables.

I am wondering if correlation analysis has to be done separately on the dataset in order to choose highly correlated predictors before applying PCA.

share|improve this question
2  
Because PCA begins with either the covariance or correlation matrix as its input, it is hard to see why there's a question about what is calculated first. Perhaps you might like to learn more about what PCA is and how it works: just click on the pca tag beneath your question and read some of our higher-voted threads on the subject. – whuber♦ Apr 12 '12 at 20:50
1  
In addition to @whuber 's excellent point, I'll point out that you seem to be confusing collinearity with correlation. They are not the same. Correlation is a property of pairs of variables; collinearity is a property of sets of variables. You can have collinearity without high correlation, if you have a lot of variables. – Peter Flom Apr 12 '12 at 22:25

1 Answer

up vote 1 down vote

Both examination of a correlation matrix and a principal components analysis will provide useful information about the linear relationships between your predictors.

In broad terms:

  • The correlation matrix will highlight bivariate relationships. It can detect pairs of variables with particularly high correlations where from a regression context you might want to remove one of the two or create a composite of the two.
  • PCA will highlight multivariate relationships (e.g., sets of highly intercorrelated variables; the degree to which the set of variables can be effectively modelled by a smaller number of composites). It can be useful in informing the creation of composites of the variables.

Also, in broad terms there are differences based on whether you are taking a theoretical or a predictive orientation to your regression problem. If you have a more theoretical orientation, conceptual reasons may influence when and how you create composites and which if any correlated predictors you drop from an analysis. If you have a predictive orientation, you will be more concerned with maximising prediction, albeit preferably some form of out-of-sample prediction. If you have a theoretical orientation, then you are more likely going to be building arguments from an overall examination of the correlation matrix and the PCA analysis.

share|improve this answer
1  
Re bullet 1: This topic has come up elsewhere, but I think it's important. Correlations among the independent variables tell you little about which ones to use for regression with another ("dependent") variable. It is possible for two IVs to be strongly correlated (with correlation $\rho$) and for one of them to have no correlation with the DV and yet the other can be substantially correlated with the DV (with correlation $\tau$). The only restriction is that $\rho^2+\tau^2\le 1$. If you don't take care, you could throw away the only IV of any value in the regression! – whuber♦ Apr 13 '12 at 13:45

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.