Basic question on PCA: can it be used to divide original features into groups?

Question

I know this is basic question, but I have trouble understanding principal component analysis (PCA).

PCA can be used for dimensionality reduction. Let's assume we have 100 features that we think are redundant. After applying PCA, we find that 2 principal components explain 90% of the variance.

1. How can we divide the 100 original features into two groups (corresponding to the 2 main PCs?).
2. Can the eigenvectors associated to the main PCs be used as 'representatives' of these two groups?

Answer 1

Consider the goal of PCA (definition from wikipedia)

Principal component analysis (PCA) is a statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components.

So PCA takes your 100 variables and gives you 100 new variables that are not correlated. Suppose too that two new variables explain 90% of the variation. Then you say that you will work with only these two. In the words, you have chosen to work with two linear transformations of your original data (the first two PCA loadings (eigenvectors)) that map your 100 variables to 2 new ones that capture most of the variation.

Now, to get to your questions:

1. It may be that the first and second loadings match to different variables. For instance, the first loading could have zero values for the last 50 original variables, and the second loading could have zeros for the first 50 original variables. Then you could say that PCA leads to an interpretation where the first and second components divide your original variables. However, this certainly need not be the case. Both loadings can, in principle, load on all of the variables so that a clean division is not obvious. Therefore, in answer to your first question, you can divide your variables when some have high loadings on one PC and the rest have high loadings on the other PC, but this need not be the case. You have to check. To check, an easy way might be to look at the bi-plot.
2. If there is a clear pattern in your loadings, then yes, in a sense you can use the eigenvectors as 'representatives.' This concept is used in Principal Component Regression, where regression is performed on the important PCs rather than the original variables. This idea is also very closely related to the motivation behind Factor Analysis.