4
$\begingroup$

I am reading in the fabulous book of "Exploratory Multivariate Analysis by Example Using R" 2nd edition by Husson, however when I came across this sentence about PCA loadings and their calculation I couldn't get its math or how to prove it in R code:

Loadings are interpreted as the coefficients of the linear combination of the initial variables from which the principal components are constructed. From a numerical point of view, the loadings are equal to the coordinates of the variables divided by the square root of the eigenvalue associated with the component.

How can loadings be calculated given the above statement in this R example from the variables divided by the square root of the eigenvalue of the principal component?

I know that each principal component is a linear combination of the variables and loadings are the coefficients of these linear combinations.
Example

A <- as.matrix(data.frame(mtcars[,c(1:7,10,11)]), nrow = 9, byrow = TRUE)
S <- scale(A)
pca_svd <- svd(S)

pca_svd$v # here is the loading matrix
        [,1]    [,2]    [,3]     [,4]   [,5]    [,6]    [,7]    [,8]    [,9]
 [1,] -0.393  0.0275 -0.2212 -0.00613 -0.321  0.7202 -0.3814 -0.1247  0.1149
 [2,]  0.403  0.0157 -0.2523  0.04070  0.117  0.2243 -0.1589  0.8103  0.1627
 [3,]  0.397 -0.0889 -0.0783  0.33949 -0.487 -0.0197 -0.1823 -0.0642 -0.6619
 [4,]  0.367  0.2694 -0.0172  0.06830 -0.295  0.3539  0.6962 -0.1657  0.2518
 [5,] -0.312  0.3417  0.1500  0.84566  0.162 -0.0154  0.0477  0.1351  0.0381
 [6,]  0.373 -0.1719  0.4537  0.19126 -0.187 -0.0838 -0.4278 -0.1984  0.5692
 [7,] -0.224 -0.4840  0.6281 -0.03033 -0.148  0.2575  0.2762  0.3561 -0.1687
 [8,] -0.209  0.5508  0.2066 -0.28238 -0.562 -0.3230 -0.0856  0.3164  0.0472
 [9,]  0.245  0.4843  0.4641 -0.21449  0.400  0.3571 -0.2060 -0.1083 -0.3205

pca_svd$d # here are the eigenvalues
[1] 13.241  8.034  3.954  2.866  2.383  1.959  1.805  1.347  0.829

sqrt(pca_svd$d) # the square root of the eigenvalues
[1] 3.639 2.834 1.988 1.693 1.544 1.400 1.343 1.161 0.911

So the A matrix has 32 rows and 9 columns (variables), so what is meant by variable coordinates and what does this statement really mean?

Update: using FactoMineR package

When I use the FactoMineR package which the above book deals with, I even get more confused as the meaning of the statement in question, see the code below:

library(FactoMineR)
res.pca <- FactoMineR::PCA(mtcars[, c(1:11)], ncp = 9, quali.sup = c(8, 9))

head(res.pca$var$coord) # here store are the coordinates of the variables 
R> head(res.pca$var$coord)
      Dim.1   Dim.2   Dim.3    Dim.4   Dim.5    Dim.6   Dim.7   Dim.8    Dim.9
mpg  -0.935  0.0397 -0.1571 -0.00315  0.1373  0.25338  0.1236 -0.0302  0.01712
cyl   0.957  0.0227 -0.1792  0.02095 -0.0501  0.07893  0.0515  0.1960  0.02423
disp  0.945 -0.1283 -0.0556  0.17477  0.2083 -0.00692  0.0591 -0.0155 -0.09860
hp    0.873  0.3888 -0.0122  0.03516  0.1261  0.12453 -0.2257 -0.0401  0.03751
drat -0.742  0.4930  0.1065  0.43535 -0.0693 -0.00541 -0.0155  0.0327  0.00567
wt    0.888 -0.2481  0.3222  0.09846  0.0802 -0.02947  0.1387 -0.0480  0.08479

# actually these are the loadings (V . Sigma) as proof to that: 
res.pca$svd$V %*% diag(res.pca$svd$vs) == res.pca$var$coord # TRUE

So how can we compute loadings according to the statement in question of FactoMineR book and package from the variable coordinates when the coordinates themselves are actually the loadings matrix as we know it ($V \cdot \Sigma$)?

Accordingly, my guess is that this statement could read like the following:

Loadings are interpreted as the coefficients of the linear combination of the initial variables from which the principal components are constructed. From a numerical point of view, the loadings are equal to the coordinates of the variables divided which are the eigenvectors scaled up by the square root of the eigenvalue associated with the component.

$\endgroup$
  • 1
    $\begingroup$ I believe you should rather be using the FactoMineR package (which is what the authors use) to understand the above statement: FactoMineR::PCA doesn't return the loadings but the coordinates of the variables on each PC. Usually, we say that the eigenvector times the square root of the eigenvalue gives the component loadings which can be interpreted as the correlation of each item with the principal component. More information in this great post by @amoeba. $\endgroup$ – chl Oct 22 at 19:31
  • $\begingroup$ On loadings and eigenvectors of PCA stats.stackexchange.com/q/143905/3277 $\endgroup$ – ttnphns Oct 24 at 15:53
  • $\begingroup$ @chl, I added the FactoMineR package code as requested, but the statement gets more confusing by the word divided by, what could be the better wording for that then? $\endgroup$ – doctorate Oct 24 at 16:03
  • $\begingroup$ @ttnphns, I am aware of that excellent post on this confusion between the two, and how prcomp() output $rotation refers to loadings when they are actually eigenvectors (direction only devoid of magnitude reflected by the variance). But would you agree with the statement made by the author? If not, how would you re-phrase it correctly? – $\endgroup$ – doctorate Oct 24 at 16:17
  • $\begingroup$ I believe the confusion comes from the fact that the authors of the FactoMineR package use terminology related to PCA "à la française" (French school of data analysis), according to which factorial coordinates reflect correlations between variables and factor axis (or components). $\endgroup$ – chl Oct 24 at 18:24
3
$\begingroup$

I'm answering only to the citation at the beginning of the question. I did not consider the analysis in R provided in the question.

I admit that the author of the first passage may have made some confusion in terminology. Let us define properly.

$v$ is the eigenvector's values. It is the eigenvector from eigencecomposition of the covariance matrix of the variables or, equivalently, the right eigenvector from SVD decomposition of the data matrix. Eigenvector values are the standardized coordinates of the variables as vectors in the space of the principal components, i.e. on a biplot or loading plot - see last paragraph here.)

$\lambda$ is the corresponding eigenvalue (the component's variance).

$a=v \sqrt{\lambda}$ is the loading vector. Loadings $a$ are the coefficients of the linear combination predicting a variable by unit-scaled principal components. $V_i= a_{i1}F_1+a_{i2}F_2...$.

While the coefficients of the linear combination to compute unit-scaled principal component values (scores), $F_j= b_{1j}V_1+b_{2j}V_2...$, are given by

$b=a/\lambda=\frac{v\sqrt{\lambda}}{\lambda}=v/\sqrt{\lambda}$

(see here, "Computation of B in PCA: ... When component loadings are extracted but not rotated (...) amounts to simply dividing each column of A by the respective eigenvalue - the component's variance").

So you have eigenvector divided by the square root of the corresponding eigenvalue. The original citation was:

Loadings are interpreted as the coefficients of the linear combination of the initial variables from which the principal components are constructed. ... the loadings are equal to the coordinates of the variables divided by the square root of the eigenvalue associated with the component.

Then, if my interpretation is correct, that passage should sound like:

Component score coefficients are the coefficients of the linear combination of the initial variables from which the principal components [component scores] are constructed. ... the coefficients are equal to the coordinates of the variables [the eigenvectors] divided by the square root of the eigenvalue associated with the component.

In this case, the author seems to mean component score coefficients by the term "loadings".

This is so if by "coordinates of the variables" the author meant eigenvectors (standardized = normalized = unit-scaled coordinates of the variables) and implied the components themselves to be unit-scaled (as we often think in factor analysis context).

But if by "coordinates of the variables" the author meant loadings (variance-scaled or full-scaled coordinates of the variables) and implied the components themselves to be variance-scaled or full-scaled (as we often think in PCA context), then we are in the situation of tautology:

Since $a=v \sqrt{\lambda}$, then $v=a/ \sqrt{\lambda}$,

and also we know that then eigenvector values are the coefficients $b$ to compute the full-scaled principal components from the input variables. If that is what the author meant, then the passage should sound like:

Eigenvectors are the component score coefficients, the coefficients of the linear combination of the initial variables from which the principal components [component scores] are constructed. ... the coefficients are equal to the coordinates of the variables [the loadings] divided by the square root of the eigenvalue associated with the component.

In this case, the author seems to mean eigenvector entries by the term "loadings".


I might recommend to read the paragraph "Digression" in my answer about the similarities and differences between PCA and Factor analysis.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ from the above R analysis, by numbers I would confirm that per FactoMineR package, the output$var$coord means loadings (variance-scaled or full-scaled coordinates of the variables) then according to your second premise we are in the situation of tautalogy... $\endgroup$ – doctorate Oct 27 at 16:22
  • $\begingroup$ can I say that the $B$ matrix would be the change of basis matrix that would transform my newly obtained data in the experiment to the newly projected PCA space spanned by the first plane for example of PC1 and PC2? So $B$ is $V$ when $XV$= $U \Sigma$ = PC scores. $\endgroup$ – doctorate Oct 27 at 16:24
  • $\begingroup$ Yes, correct. PC scores (unstandardized, i.e. with their native variance) are computed directly by eigenvectors, so B=V then. $\endgroup$ – ttnphns Oct 27 at 18:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.