3
$\begingroup$

I am investigating many different kinds of PCA versions, I am trying to find out whether PCR will apply to my analysis thus the question on use of PCR.

$\endgroup$
  • 3
    $\begingroup$ It would be very helpful to add some context about your particular dataset and what you want to do in your analysis. PCA can be viewed as a dimensionality reduction method (helpful in a descriptive purpose) or a way to construct linear combinations of the original variables that can be subsequently used in a modeling framework, but as your question stands it is difficult to provide an answer. $\endgroup$ – chl May 25 '11 at 10:02
  • $\begingroup$ I am interested in projecting my variables onto the eigenvectors yet I am not sure whether I want to compute dot product, correlate with eigenvectors. I came across some papers that did regression with eigenvectors at this stage, this is where my question stems from $\endgroup$ – JIGsawed May 25 '11 at 12:38
4
$\begingroup$

When doing a PCA, you are effectively choosing a new set of 'variables' that you know for all your observations. Their main property is that they maximize the variance-content in one dimension (first PC has the most,...), while being linear combinations of the original covariates. This is the way it works like a dimension reduction: if 3 PCs contain 99% of the variance delivered by 100 covariates, there is not much reason, it seems, to keep the 100 covariates.

PCR essentially does regression on a set of principal components. Initially it makes sense, and in quite a few cases it does work.

However, in this regard, it is useful to look at Fisher's interpretation of discriminant analysis: he poses the problem as finding the direction(s) where the between-classes variance is maximal wrt the within-class variance.

This is where PCA fails somewhat (or could fail): it finds the direction where the 'overall' variance in the covariates is maximal (a much simpler problem), and then hopes this discriminates well. So, there is some criticism on the method, but that must not stop it from working :-)

In general, doing a clustering style algorithm on your covariates first, and then using the results for classification is not a practice I'd recommend: perhaps the strongest structure in the covariates alone is not the most efficient one for prediction of another variable.

$\endgroup$
1
$\begingroup$

The classic problem with PCR is that principal components corresponding to small eigenvalues (and hence discarded) can be significant for explaining the dependent variable. One of the solutions to this problem is to use PLS regression. In PLS regression the principal components are picked to have maximal correlation with dependent variable.

$\endgroup$
  • 2
    $\begingroup$ (+1) Note that that the constraint is one of maximal covariance, not correlation, in case of PLS regression. $\endgroup$ – chl May 27 '11 at 7:47
  • $\begingroup$ @mpiktas, chl was right. Do you maybe want to edit and replace "correlation" with "covariance"? Also, it would be more accurate to refer to PLS components as "components" instead of "principal components" (this is reserved for PCA). $\endgroup$ – amoeba Feb 7 '15 at 1:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.