Which variables explain which PCA components, and vice versa?

Question

Using this data:

head(USArrests)
nrow(USArrests)

I can do a PCA as thus:

plot(USArrests)
otherPCA <- princomp(USArrests)

I can get the new components in

otherPCA$scores

and the proportion of variance explained by components with

summary(otherPCA)

But what if I want to know which variables are mostly explained by which principal components? And vice versa: is e.g. PC1 or PC2 mostly explained by murder? How can I do this?

Can I say for instance that PC1 is 80% explained by murder or assault?

I think the loadings help me here, but they show the directionality not the variance explained as i understand it, e.g.

otherPCA$loadings

Loadings:
         Comp.1 Comp.2 Comp.3 Comp.4
Murder                         0.995
Assault  -0.995                     
UrbanPop        -0.977 -0.201       
Rape            -0.201  0.974   

Answer 1

You are right, the loadings can help you here. They can be used to compute the correlation between the variables and the principal components. Moreover, the sum of the squared loadings of one variable over all principal components is equal to 1. Hence, the squared loadings tell you the proportion of variance of one variable explained by one principal component.

The problem with princomp is, it only shows the "very high" loadings. But since the loadings are just the eigenvectors of the covariance matrix, one can get all loadings using the eigen command in R:

 loadings <- eigen(cov(USArrests))$vectors
 explvar <- loadings^2

Now, you have the desired information in the matrix explvar.

Answer 2

I think that the accepted answer can be dangerously misleading (-1). There are at least four different questions mixed together in the OP. I will consider them one after another.

---

• Q1. How much of the variance of a given PC is explained by a given original variable? How much of the variance of a given original variable is explained by a given PC?

These two questions are equivalent and the answer is given by the square $r^2$ of the correlation coefficient between the variable and the PC. If PCA is done on the correlations, then the correlation coefficient $r$ is given (see here) by the corresponding element of the loadings. PC $i$ is associated with an eigenvector $\mathbf V_i$ of the correlation matrix and the corresponding eigenvalue $s_i$. A loadings vector $\mathbf L_i$ is given by $\mathbf L_i = (s_i)^{1/2} \mathbf V_i$. Its elements are correlations of this PC with the respective original variables.

Note that eigenvectors $\mathbf V_i$ and loadings $\mathbf L_i$ are two different things! In R, eigenvectors are confusingly called "loadings"; one should be careful: their elements are not the desired correlations. [The currently accepted answer in this thread confuses the two.]

In addition, if PCA is done on covariances (and not on correlations), then loadings will also give you covariances, not correlations. To obtain correlations, one needs to compute them manually, following PCA. [The currently accepted answer is unclear about that.]

• Q2. How much of the variance of a given original variable is explained by a given subset of PCs? How to select this subset to explain e.g. $80\%$ of the variance?

Because PCs are orthogonal (i.e. uncorrelated), one can simply add up individual $r^2$ values (see Q1) to get the global $R^2$ value.

To select a subset, one can add PCs with the highest correlations ($r^2$) with a given original variable until the desired amount of explained variance ($R^2$) is reached.

• Q3. How much of the variance of a given PC is explained by a given subset of original variables? How to select this subset to explain e.g. $80\%$ of the variance?

An answer to this question is not automatically given by PCA! E.g. if all original variables are very strongly inter-correlated with pairwise $r=0.9$, then correlations between the first PC and all the variables will be around $r=0.9$. One cannot add these $r^2$ numbers to compute the proportion of variance of this PC explained by, say, five original variables (this would result in a nonsensical result $R^2 = 0.9\cdot0.9\cdot5>1$). Instead, one would need to regress this PC on these variables and obtain the multiple $R^2$ value.

How to select a subset explaining given amount of variance, was suggested by @FrankHarrell (+1).

Answer 3

You can do a backwards or forwards stepwise variable selection predicting a component or a linear combination of components from their constituent variables. The $R^2$ will be 1.0 at the first step if you use backwards stepdown. Even though stepwise regression is pretty much of a disaster when predicting $Y$ it can work well when the prediction is mechanistic as is the case here. You can add or remove variables until you explain 0.8 or 0.9 (for example) of the information in the principal components.

Answer 4

The US arrests data bundled with R are just an example here, but I note that the loadings calculations in the question come from a PCA of the covariance matrix. That's somewhere between arbitrary and nonsensical, as the variables are measured on different scales.

Urban population looks like a percent. California is 91% and highest.

The three crime variables appear to be number of arrests for crimes expressed relative to population size (presumably for some time period). Presumably it's documented somewhere whether it's arrests per 1000 or 10000 or whatever.

The mean of the assault variable in the given units is about 171 and the mean murder is about 8. So, the explanation of your loadings is that in large part the pattern is an artefact: it depends on the very different variability of the variables.

So, although there is sense in the data in that there are many more arrests for assaults than for murders, etc., that known (or unsurprising) fact dominates the analysis.

This shows that, as any where else in statistics, you have to think about what you are doing in a PCA.

If you take this further:

1. I'd argue that percent urban is better left out of the analysis. It's not a crime to be urban; it might of course serve proxy for variables influencing crime.

2. A PCA based on a correlation matrix would make more sense in my view. Another possibility is to work with logarithms of arrest rates, not arrest rates (all values are positive; see below).

Note: @random_guy's answer deliberately uses the covariance matrix.

Here are some summary statistics. I used Stata, but that's quite immaterial.

    Variable |       Obs        Mean    Std. Dev.       Min        Max
-------------+--------------------------------------------------------
   urban_pop |        50       65.54    14.47476         32         91
      murder |        50       7.788     4.35551         .8       17.4
        rape |        50      21.232    9.366384        7.3         46
     assault |        50      170.76    83.33766         45        337