Explanation of a loading plot of PCA or Factor analysis.
Loading plot shows variables as points in the space of principal components (or factors). The coordinates of variables are, usually, the loadings. (If you properly combine loading plot with the corresponding scatterplot of data cases in the same components space, that would be biplot.)
Let us have 3 somehow correlated variables, $V$, $W$, $U$. We center them and perform PCA, extracting 2 first principal components out of three: $F_1$ and $F_2$. We use loadings as the coordinates to do the loading plot below. Loadings are the unstandardized eigenvectors' elements, i.e. eigenvectors endowed by corresponding component variances, or eigenvalues.
Loading plot is the plane on the picture. Let's consider only variable $V$. The arrow habitually drawn on a loading plot is what is labeled $h'$ here; the coordinates $a_1$, $a_2$ are the loadings of $V$ with $F_1$ and $F_2$, respectively (please know that terminologically is more correct to say "component loads a variable", not vice versa).
Arrow $h'$ is the projection, on the component plane, of vector $h$ which is the true position of variable $V$ in the variables' space spanned by $V$, $W$, $U$. The squared length of the vector, $h^2$, is the variance$\bf^a$ of $V$. While $h'^2$ is the portion of that variance explained by the two components.
Loading, correlation, projected correlation. Since variables were centered prior extraction of components, $\cos \phi$ is the Pearson correlation between $V$ and component $F_1$. That should not be confused with $\cos \alpha$ on the loading plot, which is another quantity: it is Pearson correlation between component $F_1$ and variable vectored here as $h'$. As a variable, $h'$ is the prediction of $V$ by the (standardized) components in linear regression (compare with drawing of linear regression geometry here) where loadings $a$'s are the regression coefficients (when components are kept orthogonal, as extracted).
Further. We may remember (trigonometry) that $a_1 = h \cdot \cos \phi$. It can be understood as the scalar product between vector $V$ and unit-length vector $F_1$: $h \cdot 1 \cdot \cos \phi$. $F_1$ is set that unit-variance vector because it has no its own variance apart from that variance of $V$ which it explains (by amount $h'$): i.e. $F_1$ is an extracted-from-V,W,U and not an invited-from-outside entity. Then, clearly, $a_1 = \sqrt{var_{V} \cdot var_{F_1}} \cdot r = h \cdot 1 \cdot \cos \phi$ is the covariance between $V$ and standardized, unit-scaled$\bf^b$ (to set $s_1=\sqrt{var_{F_1}}=1$) component $F_1$. This covariance is directly comparable with the covariances between the input variables; for example, the covariance between $V$ and $W$ will be the product of their vector lengths multiplied by the cosine between them.
To sum up: loading $a_1$ can be seen as the covariance between the standardized component and the observed variable, $h \cdot 1 \cdot \cos \phi$, or equivalently between the standardized component and the explained (by all the components defining the plot) image of the variable, $h' \cdot 1 \cdot \cos \alpha$. That $\cos \alpha$ could be called V-F1 correlation projected on the F1-F2 component subspace.
The aforesaid correlation between a variable and a component, $\cos \phi = a_1/h$, is also called standardized or rescaled loading. It is convenient in interpretation of components because it in the range [-1,1].
Relation to eigenvectors. Rescaled loading $\cos \phi$ should not be confused with the eigenvector element which - as we know it - is the cosine of the angle between a variable and a principal component. Recall that loading is eigenvector element scaled up by the component's singular value (sq. root of the eigenvalue). I.e. for variable $V$ of our plot: $a_1= e_1s_1$, where $s_1$ is the st. deviation (not $1$ but original, i.e. the singular value) of $F_1$ latent variable. Then it comes that eigenvector element $e_1= \frac{a_1}{s_1}=\frac{h}{s_1}\cos \phi$, not the $\cos \phi$ itself. The confusion around two words "cosine" dissolves when we recall what kind of space representation we are in. Eigenvector value is cosine of the angle of rotation of a variable as axis into pr. component as axis within variable space (aka scatterplot view), such as here. While $\cos \phi$ on our loading plot is the cosine similarity measure between a variable as vector and a pr. component as ... well.. as vector too, if you like (albeit it is drawn as axis on the plot), - for we are currently in the subject space (which loading plot is) where correlated variables are fans of vectors - not are orthogonal axes, - and the vector angles are the measure of association - and not of space base rotation.
Whereas loading is the angular (i.e. scalar product type) association measure between a variable and a unit-scaled component, and rescaled loading is the standardized loading where the scale of the variable is reduced to unit either, but eigenvector coefficient is the loading where the component is "overstandardized", i.e. was brought to scale $1/s$ (rather than 1); alternatively, it can be thought of as a rescaled loading where scale of the variable was brought to $h/s$ (instead of 1).
So, what are associations between a variable and a component? You may choose what you like. It may be the loading (covariance with unit scaled component) $a$; the rescaled loading $\cos \phi$ (= variable-component correlation); correlation between the image (prediction) and the component (= projected correlation $\cos \alpha$). You might even choose eigenvector coefficient $e= a/s$ if you need (though I wonder what might be a reason). Or invent your own measure.
Eigenvector value squared has the meaning of the contribution of a variable into a pr. component. Rescaled loading squared has the meaning of the contribution of a pr. component into a variable.
Relation to PCA based on correlations. If we PCA-analyzed not just centered but standardized (centered then unit-variance scaled) variables, then the three variables vectors (not their projections on the plane) would be of the same, unit length. Then it automatically follows that a loading is correlation, not covariance, between a variable and a component. But that correlation won't be equal to "standardized loading" $\cos \phi$ of the picture above (based on the analysis of just centered variables), because PCA of standardized variables (correlations-based PCA) yields different components than PCA of centered variables (covariances-based PCA). In correlation-based PCA $a_1= \cos \phi$ because $h=1$, but principal components are not those same principal components as we get from covariances-based PCA (read, read).
In factor analysis, loading plot has basically the same concept and interpretation as in PCA. The only (but important) difference is the substance of $h'$. In factor analysis, $h'$ - called then "communality" of the variable - is the portion of its variance that is explained by common factors which are responsible specifically for correlations among variables. While in PCA the explained portion $h'$ is gross "mixture" - it partly represents correlatedness and partly unrelatedness among variables. With factor analysis, the plane of loadings on our picture would be oriented differently (actually, it will even extend out of our 3d variables' space into the 4th dimension, which we cannot draw; the loadings plane won't be a subspace of our 3d space spanned by $V$ and the other two variables), and projection $h'$ will be of another length and with another angle $\alpha$. (The theoretical difference between PCA and factor analysis is explained geometrically here via subject space representation and here via variable space representation.)
$\bf^{a,b}$ A reply to @Antoni Parellada's request in comments. It is equivalent whether you prefer to speak in terms of variance or in terms of scatter (SS of deviation): variance = scatter $/(n-1)$, where $n$ is the sample size. Because we are dealing with one dataset with same $n$, the constant changes nothing in the formulas. If $\bf X$ is the data (with variables V,W,U centered), then the eigendecomposition of its (A) covariance matrix yields same eigenvalues (component variances) and eigenvectors as the eigendecomposition of (B) scatter matrix $\bf X'X$ obtained after initial division of $\bf X$ by $\sqrt{n-1}$ factor. After that, in the formula of a loading (see the middle section of the answer), $a_1 = h \cdot s_1 \cdot \cos \phi$, term $h$ is st. deviation $\sqrt{var_{V}}$ in (A) but root scatter (i.e. norm) $\Vert V \Vert$ in (B). Term $s_1$, which equals $1$, is the standardized $F_1$ component's st. deviation $\sqrt{var_{F_1}}$ in (A) but root scatter $\Vert F_1 \Vert$ in (B). Finally, $\cos \phi = r$ is the correlation which is insensitive to the usage of $n-1$ in its calculations. Thus, we simply speak conceptually of variances (A) or of scatters (B), while the values themselves remain the same in the formula in both instances.
](https://i.sstatic.net/pyoa7.png)
