7
$\begingroup$

In a PCA, the loadings can be understood as the weights for each original variable when calculating the principal component, or "how much each variable influence a principal component".

Thus, can we consider the loading value of each variable as how correlated the original variable is with the respective principal component? If a variable has a high loading value in a principal component, it necessarily has a high correlation with that same component? And if this variable presents a loading value of 1/-1, will it necessarily present a correlation of 1/-1 with the same principal component?

$\endgroup$
1
  • $\begingroup$ sure. The loadings do tell you something about the direction of the correlation. $\endgroup$
    – utobi
    Jan 14, 2023 at 20:03

1 Answer 1

8
$\begingroup$

You can answer the questions yourself if you look at how the PCA is defined. For this, let $\mathbb{X}$ denote the $n\times p$ data matrix, and let $S = [s_{ij}]$ be the sample covariance matrix, e.g. $S = (n-1)^{-1} (\mathbb{X}^\top H \mathbb{X})$, where $H = I_n - \frac{1}{b}1_n1_n^\top$ is the centering matrix.

For simplicity, let's assume $\mathbb{X}$ has full rank. Consider the spectral decomposition of $S$, $$S = \Gamma\Lambda \Gamma^\top,$$ where $\Gamma = [\gamma_1|\cdots|\gamma_p]$, $\Lambda = \text{diag}(\lambda_1,\ldots,\lambda_p)$, with $\gamma_i$ the $i$th eigenvector associated to the $i$the eigen value $\lambda_i$. The $i$th principal component (PC) is defined as

$$ y_i = X\gamma_i, $$

and the sample product moment correlation between the $i$the PC and, say, $X_{j}$ the $j$th variable (i.e. the $j$the column of $\mathbb{X}$) is

$$r_{y_i, X_j} = \frac{S_{y_i,X_j}}{\sqrt{S_{y_i}^2 S_{X_j}^2}},$$

where $S_{y_i, X_j}$ is the sample covariance between $y_i$ and $X_j$. With some algebra, it is possible to show that

\begin{align*} r_{y_i, X_j} = \frac{\gamma_{ij}\sqrt{\lambda_i}}{s_{kk}}, \text{for all }i,j\in\{1,\ldots,p\},\tag{*} \end{align*}

where $\gamma_{ij}$ is the $j$the element of $\gamma_{i}$ and $s_{kk}$ is the standard deviation of $X_j$.

As you can see from (*), both the sign and the magnitude of the correlation are related to the loadings of the PCA, e.g. the eigenvectors of $S$. However, the correlation itself depends also on the variance of $X_j$ and on the $i$th eigenvalue. Thus a loading of $\pm 1$ doesn't necessarily imply a correlation equal to $\pm 1$. However, the higher the loading the higher the correlation, ceteris paribus.

Be careful when you interpret the results of a PCA, because the sign of the correlation is arbitrary, because if $\gamma_i$ is an eigenvector of $S$, then $-\gamma_i$ is also a valid eigenvector. When interpreting the PCA, indeed, we care about the sign and magnitude of loadings of a variable relative to others. For instance, if $X_1$ and $X_2$ have got high loadings with a different sign, then we can say that they are correlated with the PC in question, with one variable being positively correlated and the other negatively correlated; but it is not meaningful to ask which one has got positive and which negative value since the sign is arbitrary.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.