How to interpret loadings and eigenvectors in PCA

Question

I have a question about loading vectors and Eigen vectors. For the purposes of this discussion, a loading vector is an Eigen vector multiplied by the square root of the Eigen value.

1. If the data is standardized (centered and divided by the standard deviation), is it correct to interpret the resulting loading vector coefficients as the correlation between the variables related to the loading vector coefficients and the loading vector in which the coefficients reside.
2. If the data on which PCA analysis is performed is centered but we do not divide by the standard deviation, is it correct to interpret the loading vector coefficients as the covariance between the variables related to the loading vector coefficients and the loading vector in which the coefficients reside.
3. If the data on which PCA analysis is performed is centered and the data is all in the same units (like TFIDF data for example), is it correct to interpret the resulting loading vector coefficients as the correlation between the variables related to the loading vector coefficients and the loading vector in which the coefficients reside.
4. In any of the above cases, if we compute Eigen vectors instead of loading vectors, is it correct to interpret the Eigen vectors as containing the cosines of orthogonal transformation (rotation) of variables into principal components.

Answer 1

1. Yes. If the variables were standardized (that is alias to say that the PCA was based on correlation matrix), then loading values are correlations between the PCs and the variables. (All the p X m loadings are the correlations.)
2. Likewise: Yes. If the variables were centered (that is alias to say that the PCA was based on covariance matrix), then loading values are covariances between the PCs and the variables.
3. No. The loadings are covariances. Because there was no standardization, only was centering, the matrix implicitly analyzed is the covariance matrix, it has variances, not $1$s, on its diagonal.
4. Yes. Eigenvector entries are the cosines of orthogonal transformation (rotation). It is so in both cases - when you analyze centered variables (covariance matrix) or standardized variables (correlation matrix). But the cosines will usually be different in the two cases because the rotation is different. That is to say, PCs are different when you base your PCA on standardized data and when you base it on centered data.