# Can we consider the loadings as a proxy for correlation, in a Principal Component Analysis (PCA)?

In a PCA, the loadings can be understood as the weights for each original variable when calculating the principal component, or "how much each variable influence a principal component".

Thus, can we consider the loading value of each variable as how correlated the original variable is with the respective principal component? If a variable has a high loading value in a principal component, it necessarily has a high correlation with that same component? And if this variable presents a loading value of 1/-1, will it necessarily present a correlation of 1/-1 with the same principal component?

• sure. The loadings do tell you something about the direction of the correlation. Jan 14, 2023 at 20:03

You can answer the questions yourself if you look at how the PCA is defined. For this, let $$\mathbb{X}$$ denote the $$n\times p$$ data matrix, and let $$S = [s_{ij}]$$ be the sample covariance matrix, e.g. $$S = (n-1)^{-1} (\mathbb{X}^\top H \mathbb{X})$$, where $$H = I_n - \frac{1}{b}1_n1_n^\top$$ is the centering matrix.

For simplicity, let's assume $$\mathbb{X}$$ has full rank. Consider the spectral decomposition of $$S$$, $$S = \Gamma\Lambda \Gamma^\top,$$ where $$\Gamma = [\gamma_1|\cdots|\gamma_p]$$, $$\Lambda = \text{diag}(\lambda_1,\ldots,\lambda_p)$$, with $$\gamma_i$$ the $$i$$th eigenvector associated to the $$i$$the eigen value $$\lambda_i$$. The $$i$$th principal component (PC) is defined as

$$y_i = X\gamma_i,$$

and the sample product moment correlation between the $$i$$the PC and, say, $$X_{j}$$ the $$j$$th variable (i.e. the $$j$$the column of $$\mathbb{X}$$) is

$$r_{y_i, X_j} = \frac{S_{y_i,X_j}}{\sqrt{S_{y_i}^2 S_{X_j}^2}},$$

where $$S_{y_i, X_j}$$ is the sample covariance between $$y_i$$ and $$X_j$$. With some algebra, it is possible to show that

\begin{align*} r_{y_i, X_j} = \frac{\gamma_{ij}\sqrt{\lambda_i}}{s_{kk}}, \text{for all }i,j\in\{1,\ldots,p\},\tag{*} \end{align*}

where $$\gamma_{ij}$$ is the $$j$$the element of $$\gamma_{i}$$ and $$s_{kk}$$ is the standard deviation of $$X_j$$.

As you can see from (*), both the sign and the magnitude of the correlation are related to the loadings of the PCA, e.g. the eigenvectors of $$S$$. However, the correlation itself depends also on the variance of $$X_j$$ and on the $$i$$th eigenvalue. Thus a loading of $$\pm 1$$ doesn't necessarily imply a correlation equal to $$\pm 1$$. However, the higher the loading the higher the correlation, ceteris paribus.

Be careful when you interpret the results of a PCA, because the sign of the correlation is arbitrary, because if $$\gamma_i$$ is an eigenvector of $$S$$, then $$-\gamma_i$$ is also a valid eigenvector. When interpreting the PCA, indeed, we care about the sign and magnitude of loadings of a variable relative to others. For instance, if $$X_1$$ and $$X_2$$ have got high loadings with a different sign, then we can say that they are correlated with the PC in question, with one variable being positively correlated and the other negatively correlated; but it is not meaningful to ask which one has got positive and which negative value since the sign is arbitrary.