I keep reading that the angle between loadings in a PCA plot indicates some degree of correlation between the loadings (presumably lower angles lead to higher correlation, and vice versa).
I don't fully understand this. Say we have two features, $X$ and $Y$, where $Y=cX + \epsilon$, that is, Y is correlated with X plus some small amount of random noise. If I run a PCA on this, I will get one large principal component and one small. If the loading vectors correspond to the features themselves, then I will simply get the projection of the original axes onto the principal components; these projected vectors don't seem to have a low angle between them.
So what exactly is the relationship between the angles between loading vectors and their degree of correlation? To me, a low angle between two loadings seems to only indicate that two particular features have the same level of contribution towards a particular direction of variance, but I'm not sure if there's more than just this.
