Loadings vs eigenvectors in PCA: when to use one or another?

Question

In principal component analysis (PCA), we get eigenvectors (unit vectors) and eigenvalues. Now, let us define loadings as $$\text{Loadings} = \text{Eigenvectors} \cdot \sqrt{\text{Eigenvalues}}.$$

I know that eigenvectors are just directions and loadings (as defined above) also include variance along these directions. But for my better understanding, I would like to know where I should use loadings instead of eigenvectors? An example would be perfect!

I have generally only seen people using eigenvectors but every once in a while they use loadings (as defined above) and then I am left feeling that I do not really understand the difference.

Answer 1

In PCA, you split covariance (or correlation) matrix into scale part (eigenvalues) and direction part (eigenvectors). You may then endow eigenvectors with the scale: loadings. So, loadings are thus become comparable by magnitude with the covariances/correlations observed between the variables, - because what had been drawn out from the variables' covariation now returns back - in the form of the covariation between the variables and the principal components. Actually, loadings are the covariances/correlations between the original variables and the unit-scaled components. This answer shows geometrically what loadings are and what are coefficients associating components with variables in PCA or factor analysis.

Loadings:

1. Help you interpret principal components or factors; Because they are the linear combination weights (coefficients) whereby unit-scaled components or factors define or "load" a variable.

   (Eigenvector is just a coefficient of orthogonal transformation or projection, it is devoid of "load" within its value. "Load" is (information of the amount of) variance, magnitude. PCs are extracted to explain variance of the variables. Eigenvalues are the variances of (= explained by) PCs. When we multiply eigenvector by sq.root of the eivenvalue we "load" the bare coefficient by the amount of variance. By that virtue we make the coefficient to be the measure of association, co-variability.)

2. Loadings sometimes are "rotated" (e.g. varimax) afterwards to facilitate interpretability (see also);

3. It is loadings which "restore" the original covariance/correlation matrix (see also this thread discussing nuances of PCA and FA in that respect);

4. While in PCA you can compute values of components both from eigenvectors and loadings, in factor analysis you compute factor scores out of loadings.

5. And, above all, loading matrix is informative: its vertical sums of squares are the eigenvalues, components' variances, and its horizontal sums of squares are portions of the variables' variances being "explained" by the components.

6. Rescaled or standardized loading is the loading divided by the variable's st. deviation; it is the correlation. (If your PCA is correlation-based PCA, loading is equal to the rescaled one, because correlation-based PCA is the PCA on standardized variables.) Rescaled loading squared has the meaning of the contribution of a pr. component into a variable; if it is high (close to 1) the variable is well defined by that component alone.

An example of computations done in PCA and FA for you to see.

Eigenvectors are unit-scaled loadings; and they are the coefficients (the cosines) of orthogonal transformation (rotation) of variables into principal components or back. Therefore it is easy to compute the components' values (not standardized) with them. Besides that their usage is limited. Eigenvector value squared has the meaning of the contribution of a variable into a pr. component; if it is high (close to 1) the component is well defined by that variable alone.

Although eigenvectors and loadings are simply two different ways to normalize coordinates of the same points representing columns (variables) of the data on a biplot, it is not a good idea to mix the two terms. This answer explained why. See also.

Answer 2

There seems to be a great deal of confusion about loadings, coefficients and eigenvectors. The word loadings comes from Factor Analysis and it refers to coefficients of the regression of the data matrix onto the factors. They are not the coefficients defining the factors. See for example Mardia, Bibby and Kent or other multivariate statistics textbooks.

In recent years the word loadings has been used to indicate the PCs coefficients. Here it seems that it used to indicate the coefficients multiplied by the sqrt of the eigenvalues of the matrix. These are not quantities commonly used in PCA. The principal components are defined as the sum of the variables weighted with unit norm coefficients. In this way the PCs have norm equal to the corresponding eigenvalue, which in turn is equal to the variance explained by the component.

It is in Factor Analysis that the factors are required to have unit norm. But FA and PCA are completely different. Rotating the PCs' coefficient is very rarely done because it destroys the optimality of the components.

In FA the factors are not uniquely defined and can be estimated in different ways. The important quantities are the loadings (the true ones) and the communalities which are used to study the structure of the covariance matrix. PCA or PLS should be used to estimate components.

Answer 3

I am a bit confused by those names, and I searched in the book named "Statistical Methods in the Atmospherical Science", and it gave me a summary of varied Terminology of PCA, here are the screenshots in the book, hope it will help.

Answer 4

There appears to be some confusion over this matter, so I will provide some observations and a pointer to where an excellent answer can be found in the literature.

Firstly, PCA and Factor Analysis (FA) are related. In general, principal components are orthogonal by definition whereas factors - the analogous entity in FA - are not. Simply put, principal components span the factor space in an arbitrary but not necessarily useful way due to their being derived from pure eigenanalysis of the data. Factors on the other hand represent real-world entities which are only orthogonal (i.e. uncorrelated or independent) by coincidence.

Say we take s observations from each of l subjects. These can be arranged into a data matrix D having s rows and l columns. D can be decomposed into a score matrix S and a loading matrix L such that D = SL. S will have s rows, and L will have l columns, the second dimension of each being the number of factors n. The purpose of factor analysis is to decompose D in such a way as to reveal the underlying scores and factors. The loadings in L tell us the proportion of each score which make up the observations in D.

In PCA, L has the eigenvectors of the correlation or covariance matrix of D as its columns. These are conventionally arranged in descending order of the corresponding eigenvalues. The value of n - i.e. the number of significant principal components to retain in the analysis, and hence the number of rows of L - is typically determined through the use of a scree plot of the eigenvalues or one of numerous other methods to be found in the literature. The columns of S in PCA form the n abstract principal components themselves. The value of n is the underlying dimensionality of the data set.

The object of factor analysis is to transform the abstract components into meaningful factors through the use of a transformation matrix T such that D = STT^-1L. (ST) is the transformed score matrix, and (T^-1L) is the transformed loading matrix.

The above explanation roughly follows the notation of Edmund R. Malinowski from his excellent Factor Analysis in Chemistry. I highly recommend the opening chapters as an introduction to the subject.

Answer 5

The loadings show the relationship between the raw data and the rotated data.
$$ {\rm Loadings} = {\rm Eigenvectors}\cdot\sqrt{\rm Eigenvalues} $$ For example, in R, raw data %*% loadings = rotated data. Try it:

d          = matrix(c(-1,-1,0,2,0,-2,0,-0,1,1),ncol = 2)
loadings   = prcomp(d)$rotation[,'PC1']
PC1_scores = d %*% loadings
PC1_scores = prcomp(d)$x[,'PC1']

prcomp returns rotation which is the matrix of variable loadings (i.e., a matrix whose columns contain the eigenvectors). The function princomp returns this in the element loadings.