PCA on correlation or covariance: does PCA on correlation ever make sense? In principal component analysis (PCA), one can choose either the covariance matrix or the correlation matrix to find the components (from their respective eigenvectors). These give different results (PC loadings and scores), because the eigenvectors between both matrices are not equal. My understanding is that this is caused by the fact that a raw data vector $X$ and its standardization $Z$ cannot be related via an orthogonal transformation. Mathematically, similar matrices (i.e. related by orthogonal transformation) have the same eigenvalues, but not necessarily the same eigenvectors.
This raises some difficulties in my mind: 


*

*Does PCA actually make sense, if you can get two different answers for the same starting data set, both trying to achieve the same thing (=finding directions of maximum variance)? 

*When using the correlation matrix approach, each variable is being standardized (scaled) by its own individual standard deviation, before calculating the PCs. How, then, does it still make sense to find the directions of maximum variance if the data have already been scaled/compressed differently beforehand? I know that that correlation based PCA is very convenient (standardized variables are dimensionless, so their linear combinations can be added; other advantages are also based on pragmatism), but is it correct?
It seems to me that covariance based PCA is the only truly correct one (even when the variances of the variables differ greatly), and that whenever this version cannot be used, correlation based PCA should not be used either.
I know that there is this thread: PCA on correlation or covariance? -- but it seems to focus only on finding a pragmatic solution, which may or may not also be an algebraically correct one.
 A: Speaking from a practical viewpoint - possibly unpopular here - if you have data measured on different scales, then go with correlation ('UV scaling' if you are a chemometrician), but if the variables are on the same scale and the size of them matters (e.g. with spectroscopic data), then covariance (centering the data only)  makes more sense. PCA is a scale-dependent method and also log transformation can help with highly skewed data. 
In my humble opinion based on 20 years of practical application of chemometrics you have to experiment a bit and see what works best for your type of data. At the end of the day you need to be able to reproduce your results and try to prove the predictability of your conclusions. How you get there is often a case of trial and error 
but the thing that matters is that what you do is documented and reproducible. 
A: I hope these responses to your two questions will calm your concern:


*

*A correlation matrix is a covariance matrix of the standardized (i.e. not just centered but also rescaled) data; that is, a covariance matrix (as if) of another, different dataset. So it is natural and it shouldn't bother you that the results differ.

*Yes it makes sense to find the directions of maximal variance with standardized data - they are the directions of - so to speak - "correlatedness," not "covariatedness"; that is, after the effect of unequal variances - of the original variables - on the shape of the multivariate data cloud was taken off.



Next text and pictures added by @whuber (I thank him. Also, see my comment below)
Here is a two-dimensional example showing why it still makes sense to locate the principal axes of standardized data (shown on the right).  Note that in the right hand plot the cloud still has a "shape" even though the variances along the coordinate axes are now exactly equal (to 1.0).  Similarly, in higher dimensions the standardized point cloud will have a non-spherical shape even though the variances along all axes are exactly equal (to 1.0).  The principal axes (with their corresponding eigenvalues) describe that shape.  Another way to understand this is to note that all the rescaling and shifting that goes on when standardizing the variables occurs only in the directions of the coordinate axes and not in the principal directions themselves.

What is happening here is geometrically so intuitive and clear that it would be a stretch to characterize this as a "black-box operation": on the contrary, standardization and PCA are some of the most basic and routine things we do with data in order to understand them.

Continued by @ttnphns
When would one prefer to do PCA (or factor analysis or other similar type of analysis) on correlations (i.e. on z-standardized variables) instead of doing it on covariances (i.e. on centered variables)?


*

*When the variables are different units of measurement. That's clear.

*When one wants the analysis to reflect just and only linear associations. Pearson r is not only the covariance between the uniscaled (variance=1) variables; it is suddenly the measure of the strength of linear relationship, whereas usual covariance coefficient is receptive to both linear and monotonic relationship.

*When one wants the associations to reflect relative co-deviatedness (from the mean) rather than raw co-deviatedness. The correlation is based on distributions, their spreads, while the covariance is based on the original measurement scale. If I were to factor-analyze patients' psychopathological profiles as assesed by psychiatrists' on some clinical questionnaire consisting of Likert-type items, I'd prefer covariances. Because the professionals are not expected to distort the rating scale intrapsychically. If, on the other hand, I were to analyze the patients' self-portrates by that same questionnaire I'd probably choose correlations. Because layman's assessment is expected to be relative "other people", "the majority" "permissible deviation" or similar implicit das Man loupe which "shrinks" or "stretches" the rating scale for one.

A: I have no time to go into a fuller description of detailed & technical aspects of the experiment I described, and clarifications on wordings (recommending, performance, optimum) would again divert us away from the real issue, which is about what type of input data the PCA can(not) / should (not) be taking. PCA operates by taking linear combinations of numbers (values of variables). Mathematically, of course, one can add any two (real or complex) numbers. But if they have been re-scaled before PCA transformation, is their linear combination (and hence to process of maximization) still meaningful to operate on?
If each variable $x_i$ has same variance $s^2$, then clearly yes, because $(x_1/s_1)+(x_2/s_2)=(x_1+x_2)/s$ is still proportional and comparable to the physical superposition of data $x_1+x_2$ itself. But if $s_1\not =s_2$, then the linear combination of standardized quantities distorts the data of the input variables to different degrees. There seems little point then to maximize the variance of their linear combination. 
In that case, PCA gives a solution for a different set of data, whereby each variable is scaled differently. If you then unstandardize afterwards (when using corr_PCA) then that may be OK and necessary; but if you just take the the raw corr_PCA solution as-is and stop there, you would obtain a mathematical solution, but not one related to the physical data. As unstandardization afterwards then seems mandatory as a minimum (i.e., 'unstretching' the axes by the inverse standard deviations), cov_PCA could have been used to begin with.
If you are still reading by now, I am impressed! For now, I finish by quoting from Jolliffe's book, p. 42, which is the part that concerns me: 'It must not be forgotten, however, that correlation matrix PCs, when re-expressed in terms of the original variables, are still linear functions of x that maximize variance with respect to the standardized variables and not with respect to the original variables.' 
If you think I am interpreting this or its implications wrongly, this excerpt may be a good focus point for further discussion.
