1
$\begingroup$

I know this is basic question, but I have trouble understanding principal component analysis (PCA).

PCA can be used for dimensionality reduction. Let's assume we have 100 features that we think are redundant. After applying PCA, we find that 2 principal components explain 90% of the variance.

  1. How can we divide the 100 original features into two groups (corresponding to the 2 main PCs?).
  2. Can the eigenvectors associated to the main PCs be used as 'representatives' of these two groups?
$\endgroup$
  • $\begingroup$ There are many threads here on PCA that can help you. Did you look at any of them? There has to be something new and different about your question to justify starting a new thread. In particular, there seems here a presumption or hope that PCA divides variables into groups. On that see e.g. stats.stackexchange.com/questions/33053/… $\endgroup$ – Nick Cox May 27 '14 at 10:18
  • 1
    $\begingroup$ @bigTree, No, PCA is generally not an adequate means to cluster features into groups. But factor analysis (as well as cluster analysis, of course) is a means for that (see pt8 in the "list" here). If V1,V2 "represent" PC1 and V3,V4 represent PC2, it doesn't follow that these 4 variables don't intercorrelate outside of these 2 PCs. But with factors, it is the other way round. $\endgroup$ – ttnphns May 27 '14 at 10:56
3
$\begingroup$

Consider the goal of PCA (definition from wikipedia)

Principal component analysis (PCA) is a statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components.

So PCA takes your 100 variables and gives you 100 new variables that are not correlated. Suppose too that two new variables explain 90% of the variation. Then you say that you will work with only these two. In the words, you have chosen to work with two linear transformations of your original data (the first two PCA loadings (eigenvectors)) that map your 100 variables to 2 new ones that capture most of the variation.

Now, to get to your questions:

  1. It may be that the first and second loadings match to different variables. For instance, the first loading could have zero values for the last 50 original variables, and the second loading could have zeros for the first 50 original variables. Then you could say that PCA leads to an interpretation where the first and second components divide your original variables. However, this certainly need not be the case. Both loadings can, in principle, load on all of the variables so that a clean division is not obvious. Therefore, in answer to your first question, you can divide your variables when some have high loadings on one PC and the rest have high loadings on the other PC, but this need not be the case. You have to check. To check, an easy way might be to look at the bi-plot.
  2. If there is a clear pattern in your loadings, then yes, in a sense you can use the eigenvectors as 'representatives.' This concept is used in Principal Component Regression, where regression is performed on the important PCs rather than the original variables. This idea is also very closely related to the motivation behind Factor Analysis.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.