11
$\begingroup$

One common thing to do when doing Principal Component Analysis (PCA) is to plot two loadings against each other to investigate the relationships between the variables. In the paper accompanying the PLS R package for doing Principal Component Regression and PLS regression there is a different plot, called the correlation loadings plot (see figure 7 and page 15 in the paper). The correlation loading, as it is explained, is the correlation between the scores (from the PCA or PLS) and the actual observed data.

It seems to me that loadings and correlation loadings are pretty similar, except that they are scaled a bit differently. A reproducible example in R, with the built in data set mtcars is as follows:

data(mtcars)
pca <- prcomp(mtcars, center=TRUE, scale=TRUE)

#loading plot
plot(pca$rotation[,1], pca$rotation[,2],
     xlim=c(-1,1), ylim=c(-1,1),
     main='Loadings for PC1 vs. PC2')

#correlation loading plot
correlationloadings <- cor(mtcars, pca$x)
plot(correlationloadings[,1], correlationloadings[,2],
     xlim=c(-1,1), ylim=c(-1,1),
     main='Correlation Loadings for PC1 vs. PC2')

loadingplot correlationloadinsplot

What is the difference in interpretation of these plots? And which plot (if any) is best to use in practice?

$\endgroup$
  • $\begingroup$ for a better view of the pca, use biplot(pca), it shows you the loading and the scores of the pca and so you can interpret it better. $\endgroup$ – Paul Jun 22 '14 at 21:07
  • 6
    $\begingroup$ R prcomp package recklessly calls eigenvectors "loadings". I advice to keep these terms separate. Loadings are eigenvectors scaled up to the respective eigenvalues. $\endgroup$ – ttnphns Jun 23 '14 at 7:02
  • 1
    $\begingroup$ Explaining geometry of a loading plot: stats.stackexchange.com/a/119758/3277 $\endgroup$ – ttnphns Oct 16 '14 at 7:25
13
$\begingroup$

Warning: R uses the term "loadings" in a confusing way. I explain it below.

Consider dataset $\mathbf{X}$ with (centered) variables in columns and $N$ data points in rows. Performing PCA of this dataset amounts to singular value decomposition $\mathbf{X} = \mathbf{U} \mathbf{S} \mathbf{V}^\top$. Columns of $\mathbf{US}$ are principal components (PC "scores") and columns of $\mathbf{V}$ are principal axes. Covariance matrix is given by $\frac{1}{N-1}\mathbf{X}^\top\mathbf{X} = \mathbf{V}\frac{\mathbf{S}^2}{{N-1}}\mathbf{V}^\top$, so principal axes $\mathbf{V}$ are eigenvectors of the covariance matrix.

"Loadings" are defined as columns of $\mathbf{L}=\mathbf{V}\frac{\mathbf S}{\sqrt{N-1}}$, i.e. they are eigenvectors scaled by the square roots of the respective eigenvalues. They are different from eigenvectors! See my answer here for motivation.

Using this formalism, we can compute cross-covariance matrix between original variables and standardized PCs: $$\frac{1}{N-1}\mathbf{X}^\top(\sqrt{N-1}\mathbf{U}) = \frac{1}{\sqrt{N-1}}\mathbf{V}\mathbf{S}\mathbf{U}^\top\mathbf{U} = \frac{1}{\sqrt{N-1}}\mathbf{V}\mathbf{S}=\mathbf{L},$$ i.e. it is given by loadings. Cross-correlation matrix between original variables and PCs is given by the same expression divided by the standard deviations of the original variables (by definition of correlation). If the original variables were standardized prior to performing PCA (i.e. PCA was performed on the correlation matrix) they are all equal to $1$. In this last case the cross-correlation matrix is again given simply by $\mathbf{L}$.

To clear up the terminological confusion: what the R package calls "loadings" are principal axes, and what it calls "correlation loadings" are (for PCA done on the correlation matrix) in fact loadings. As you noticed yourself, they differ only in scaling. What is better to plot, depends on what you want to see. Consider a following simple example:

Biplots

Left subplot shows a standardized 2D dataset (each variable has unit variance), stretched along the main diagonal. Middle subplot is a biplot: it is a scatter plot of PC1 vs PC2 (in this case simply the dataset rotated by 45 degrees) with rows of $\mathbf{V}$ plotted on top as vectors. Note that $x$ and $y$ vectors are 90 degrees apart; they tell you how the original axes are oriented. Right subplot is the same biplot, but now vectors show rows of $\mathbf{L}$. Note that now $x$ and $y$ vectors have an acute angle between them; they tell you how much original variables are correlated with PCs, and both $x$ and $y$ are much stronger correlated with PC1 than with PC2. I guess that most people most often prefer to see the right type of biplot.

Note that in both cases both $x$ and $y$ vectors have unit length. This happened only because the dataset was 2D to start with; in case when there are more variables, individual vectors can have length less than $1$, but they can never reach outside of the unit circle. Proof of this fact I leave as an exercise.

Let us now take another look at the mtcars dataset. Here is a biplot of the PCA done on correlation matrix:

mtcars pca biplot

Black lines are plotted using $\mathbf{V}$, red lines are plotted using $\mathbf{L}$.

And here is a biplot of the PCA done on the covariance matrix:

mtcars pca biplot

Here I scaled all the vectors and the unit circle by $100$, because otherwise it would not be visible (it is a commonly used trick). Again, black lines show rows of $\mathbf{V}$, and red lines show correlations between variables and PCs (which are not given by $\mathbf{L}$ anymore, see above). Note that only two black lines are visible; this is because two variables have very high variance and dominate the mtcars dataset. On the other hand, all red lines can be seen. Both representations convey some useful information.

P.S. There are many different variants of PCA biplots, see my answer here for some further explanations and an overview: Positioning the arrows on a PCA biplot. The prettiest biplot ever posted on CrossValidated can be found here.

$\endgroup$
  • 2
    $\begingroup$ Although this is a very good answer (+1), it has one just didactical weakness, in that it initially puts variables in rows of X, not in columns of X as would traditionally go in statistical datasets/examples. Because of that transpose, U vectors become in the answer to be about variables and V about cases. Most people knowing PCA are accustomed to the opposite layout; so it hinders perception, a bit. $\endgroup$ – ttnphns Oct 16 '14 at 12:43
  • 1
    $\begingroup$ I might recommend to stress verbally the "moral" of the difference between the "axes biplot" and the "loadings biplot" on the scan. In the first, variability (=scale, =magnidute, =inertia, =mass) isn't presented: it is stored away in eigenvalues. In the second, it was given up fully to eigenvectors representing variables; by virtue of that "reviving" the variables become meaningful data cloud of two points, or vectors, with specific lenghths from the origin and specific angle. This is how we "suddenly" find ourselves in subject space. $\endgroup$ – ttnphns Oct 16 '14 at 13:01
  • $\begingroup$ Thanks @ttnphns, both good points. Regarding rows/columns of $\mathbf X$: in fact, I prefer the layout I used. A single data point is usually written as a column vector $\mathbf x$. A matrix $\mathbf U$ acting on it would be written as $\mathbf U \mathbf x$. If now $\mathbf X$ is a collection of column vectors stacked together, then I can write $\mathbf U\mathbf X$, which is convenient. If, instead, $\mathbf X$ has samples in rows, as you advocate, then I would need to write $\mathbf X \mathbf U^\top$, which looks weird. But I admit that many textbooks use this convention (I am not sure why). $\endgroup$ – amoeba says Reinstate Monica Oct 16 '14 at 14:44
  • 1
    $\begingroup$ This is, of course, a matter of taste. Mind, however, that the great majority of statistical programs show data spreadsheets as cases X variables. By tradition then, linear algebra in most statistical analysis texts make case a row vector. Maybe in machine learning it is different? $\endgroup$ – ttnphns Oct 16 '14 at 15:01
  • 1
    $\begingroup$ @user_anon No, this answer considers standard PCA, without any factor rotations. $\endgroup$ – amoeba says Reinstate Monica Sep 20 '18 at 20:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.