Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Using this data:

head(USArrests)
nrow(USArrests)

I can do a PCA as thus:

plot(USArrests)
otherPCA <- princomp(USArrests)

I can get the new components in

otherPCA$scores

and the proportion of variance explained by components with

summary(otherPCA)

But what if I want to know which variables are most explained by what axis. i.e. is PCA1 or PCA2 mostly explained by murder, how can do do this?

Can I say for instance PCA1 is 80% explained by murder or assault?

I think the loadings help me here, but they show the directionality not the variance explained as i understand it, e.g.

otherPCA$loadings

Loadings:
         Comp.1 Comp.2 Comp.3 Comp.4
Murder                         0.995
Assault  -0.995                     
UrbanPop        -0.977 -0.201       
Rape            -0.201  0.974   
share|improve this question
2  
Note that the signs of the loadings are arbitrary. The three crime variables are all positively correlated with each other, but you would be smart to work that from the signs of the loadings above. –  Nick Cox Dec 3 at 19:53

3 Answers 3

up vote 6 down vote accepted

You are right, the loadings can help you here. They can be used to compute the correlation between the variables and the principal components. Moreover, the sum of the squared loadings of one variable over all principal components is equal to 1. Hence, the squared loadings tell you the proportion of variance of one variable explained by one principal component.

The problem with princomp is, it only shows the "very high" loadings. But since the loadings are just the eigenvectors of the covariance matrix, one can get all loadings using the eigen command in R:

 loadings <- eigen(cov(USArrests))$vectors
 explvar <- loadings^2

Now, you have the desired information in the matrix explvar.

share|improve this answer
    
thank you random guy, could you possibly show me for example assault or urban pop we could do this? partly confused because there is only one correlation present in the matrix for assault –  user1320502 Feb 18 at 16:10
1  
Sorry, I improved my answer and did not notice you commented my post already. assault loads with -0.995 on PC1. Thus, one can conclude after squaring this value PC1 explains 99% of the variance of the variable assault. After squaring the values of urban pop, you can conclude PC3 explains 4% and PC2 95.5% of the variance of urban pop. –  random_guy Feb 18 at 16:24
    
If by "loadings" you mean eigenvectors of the covariance matrix, then they are NOT the correlations between the original variables and the PCs. Let $X$ be standardized data matrix with samples in rows, then SVD decomposition $X=USV^\top$ yields PCs in $US$ and eigenvectors of the correlation matrix in $V$ (indeed, correlation matrix is $X^\top X/N = V\frac{S^2}{N}V^\top$). Correlation between $X$ and PCs is equal to the covariance between $X$ and standardized PCs $\sqrt{N}U$, so we get $X^\top(\sqrt{N}U)/N = \sqrt{N}VS$, if I am not mistaken. In any case, I think $V$ needs to be scaled. –  amoeba Dec 3 at 17:59
    
Doesn't OP ask about how much of the PCA can be attributed to a variable? Your answer is about how much of a variable can be explained by a CPA –  Heisenberg Dec 3 at 18:40
    
@amoeba Of course, they are not the correlations. Thanks for pointing that out. –  random_guy Dec 3 at 18:51

You can do a backwards or forwards stepwise variable selection predicting a component or a linear combination of components from their constituent variables. The $R^2$ will be 1.0 at the first step if you use backwards stepdown. Even though stepwise regression is pretty much of a disaster when predicting $Y$ it can work well when the prediction is mechanistic as is the case here. You can add or remove variables until you explain 0.8 or 0.9 (for example) of the information in the principal components.

share|improve this answer
    
Neat idea--thank you. –  rolando2 Dec 4 at 0:21

The US arrests data bundled with R are just an example here, but I note that the loadings calculations in the question come from a PCA of the covariance matrix. That's somewhere between arbitrary and nonsensical, as the variables are measured on different scales.

Urban population looks like a percent. California is 91% and highest.

The three crime variables appear to be number of arrests for crimes expressed relative to population size (presumably for some time period). Presumably it's documented somewhere whether it's arrests per 1000 or 10000 or whatever.

The mean of the assault variable in the given units is about 171 and the mean murder is about 8. So, the explanation of your loadings is that in large part the pattern is an artefact: it depends on the very different variability of the variables.

So, although there is sense in the data in that there are many more arrests for assaults than for murders, etc., that known (or unsurprising) fact dominates the analysis.

This shows that, as any where else in statistics, you have to think about what you are doing in a PCA.

If you take this further:

  1. I'd argue that percent urban is better left out of the analysis. It's not a crime to be urban; it might of course serve proxy for variables influencing crime.

  2. A PCA based on a correlation matrix would make more sense in my view. Another possibility is to work with logarithms of arrest rates, not arrest rates (all values are positive; see below).

Note: @random_guy's answer deliberately uses the covariance matrix.

Here are some summary statistics. I used Stata, but that's quite immaterial.

    Variable |       Obs        Mean    Std. Dev.       Min        Max
-------------+--------------------------------------------------------
   urban_pop |        50       65.54    14.47476         32         91
      murder |        50       7.788     4.35551         .8       17.4
        rape |        50      21.232    9.366384        7.3         46
     assault |        50      170.76    83.33766         45        337
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.