37
$\begingroup$

In principal component analysis (PCA), one can choose either the covariance matrix or the correlation matrix to find the components (from their respective eigenvectors). These give different results (PC loadings and scores), because the eigenvectors between both matrices are not equal. My understanding is that this is caused by the fact that a raw data vector $X$ and its standardization $Z$ cannot be related via an orthogonal transformation. Mathematically, similar matrices (i.e. related by orthogonal transformation) have the same eigenvalues, but not necessarily the same eigenvectors.

This raises some difficulties in my mind:

  1. Does PCA actually make sense, if you can get two different answers for the same starting data set, both trying to achieve the same thing (=finding directions of maximum variance)?

  2. When using the correlation matrix approach, each variable is being standardized (scaled) by its own individual standard deviation, before calculating the PCs. How, then, does it still make sense to find the directions of maximum variance if the data have already been scaled/compressed differently beforehand? I know that that correlation based PCA is very convenient (standardized variables are dimensionless, so their linear combinations can be added; other advantages are also based on pragmatism), but is it correct?

It seems to me that covariance based PCA is the only truly correct one (even when the variances of the variables differ greatly), and that whenever this version cannot be used, correlation based PCA should not be used either.

I know that there is this thread: PCA on correlation or covariance? -- but it seems to focus only on finding a pragmatic solution, which may or may not also be an algebraically correct one.

Improve this question
$\endgroup$
8
  • 7
    $\begingroup$ I'm going to be honest and tell you I quit reading your question at some point. PCA makes sense. Yes, the results may be different depending on whether you choose to use the correlation or variance/covariance matrix. Correlation based PCA is preferred if your variables are measured on different scales, but you don't want this to dominate the outcome. Imagine if you have a series of variables that range from 0 to 1 and then some that have very large values (relatively speaking, like 0 to 1000), the large variance associated with the second group of variables will dominate. $\endgroup$
    – Patrick
    Commented Jun 27, 2013 at 0:30
  • 4
    $\begingroup$ But that's the case with many other techniques as well and I think Patrick's point is reasonable. Also it was merely a comment, no need to become aggressive. Generally speaking, why would you assume that there should be one true “algebraically” correct way to approach the problem? $\endgroup$
    – Gala
    Commented Jun 27, 2013 at 9:55
  • 7
    $\begingroup$ Perhaps you're thinking of PCA in the wrong way: it's just a transformation, so there's no question of its being correct or incorrect, or relying on assumptions about the data model - unlike, say, regression or factor analysis. $\endgroup$
    – Scortchi ♦
    Commented Jun 27, 2013 at 11:37
  • 6
    $\begingroup$ The crux of this matter appears to rest on a misunderstanding of what standardization does and how PCA works. This is understandable, because a good grasp of PCA requires visualization of higher-dimensional shapes. I would maintain that this question, like many other questions based on some sort of misapprehension, is thereby a good one and ought to remain open, because its answer(s) can reveal truths that many people might not have fully appreciated before. $\endgroup$
    – whuber ♦
    Commented Jun 27, 2013 at 14:36
  • 8
    $\begingroup$ PCA does not “claim” anything. People make claims about PCA and in fact use it very differently depending on the field. Some of these uses might be silly or questionable but it does not seem very enlightening to assume that a single variant of the technique must be the “algebraically correct” one with no reference to the context or goal of the analysis. $\endgroup$
    – Gala
    Commented Jun 27, 2013 at 20:54

3 Answers 3

Reset to default
38
$\begingroup$

I hope these responses to your two questions will calm your concern:

  1. A correlation matrix is a covariance matrix of the standardized (i.e. not just centered but also rescaled) data; that is, a covariance matrix (as if) of another, different dataset. So it is natural and it shouldn't bother you that the results differ.
  2. Yes it makes sense to find the directions of maximal variance with standardized data - they are the directions of - so to speak - "correlatedness," not "covariatedness"; that is, after the effect of unequal variances - of the original variables - on the shape of the multivariate data cloud was taken off.

Next text and pictures added by @whuber (I thank him. Also, see my comment below)

Here is a two-dimensional example showing why it still makes sense to locate the principal axes of standardized data (shown on the right). Note that in the right hand plot the cloud still has a "shape" even though the variances along the coordinate axes are now exactly equal (to 1.0). Similarly, in higher dimensions the standardized point cloud will have a non-spherical shape even though the variances along all axes are exactly equal (to 1.0). The principal axes (with their corresponding eigenvalues) describe that shape. Another way to understand this is to note that all the rescaling and shifting that goes on when standardizing the variables occurs only in the directions of the coordinate axes and not in the principal directions themselves.

Figure

What is happening here is geometrically so intuitive and clear that it would be a stretch to characterize this as a "black-box operation": on the contrary, standardization and PCA are some of the most basic and routine things we do with data in order to understand them.

Continued by @ttnphns

When would one prefer to do PCA (or factor analysis or other similar type of analysis) on correlations (i.e. on z-standardized variables) instead of doing it on covariances (i.e. on centered variables)?

  1. When the variables are different units of measurement. That's clear.
  2. When one wants the analysis to reflect just and only linear associations. Pearson r is not only the covariance between the uniscaled (variance=1) variables; it is suddenly the measure of the strength of linear relationship, whereas usual covariance coefficient is receptive to both linear and monotonic relationship.
  3. When one wants the associations to reflect relative co-deviatedness (from the mean) rather than raw co-deviatedness. The correlation is based on distributions, their spreads, while the covariance is based on the original measurement scale. If I were to factor-analyze patients' psychopathological profiles as assesed by psychiatrists' on some clinical questionnaire consisting of Likert-type items, I'd prefer covariances. Because the professionals are not expected to distort the rating scale intrapsychically. If, on the other hand, I were to analyze the patients' self-portrates by that same questionnaire I'd probably choose correlations. Because layman's assessment is expected to be relative "other people", "the majority" "permissible deviation" or similar implicit das Man loupe which "shrinks" or "stretches" the rating scale for one.
Improve this answer
$\endgroup$
16
  • 1
    $\begingroup$ 1. Sorry, but this bothers a lot. To an external individual, the standardization is a black-box operation, part of the PCA pre-conditioning of data (also in ICA). He wants one answer for his (raw) input data, especially if it relates to physical (dimensioned) data for which the PCA output needs to be interpreted physically (i.e., in terms of unstandardized variables) as well. $\endgroup$
    – Lucozade
    Commented Jun 27, 2013 at 9:29
  • 1
    $\begingroup$ Your latest revision appears to be a re-assertion that "covariance based PCA is the only truly correct one". As the entirety of the responses so far are in essence "No; wrong way to think about it; and here's why" it is difficult to know how you expect to steer discussion against such overwhelming disagreement. $\endgroup$
    – Nick Cox
    Commented Jun 28, 2013 at 10:22
  • 4
    $\begingroup$ @Lucozade: I was confused about your description of your application:- How is PCA recommending anything? How did you measure performance? Similarly for your last comment:- The optimum for what? $\endgroup$
    – Scortchi ♦
    Commented Jun 28, 2013 at 11:56
  • 5
    $\begingroup$ @Lucozade: Indeed, listen please what Scortchi said, you seem to continue chasing down spooks. PCA is simply a special form of rotating data in space. It always does optimally what it does with the input data. The cov-corr dilemma is a pragmatic one, rooted in data pre-processing and being solved at that level, not at the PCA level. $\endgroup$
    – ttnphns
    Commented Jun 28, 2013 at 12:25
  • 1
    $\begingroup$ @Lucozade: It would be my (non-expert) opinion based on your reply to me that in your specific need, you are right to want cov-based PCA. Again, your variables are all homogeneous in terms of data/measurement type (same machine type, and all data in volts). To me your example is clearly a case where cov-PCA is correct, but please note that this is not always the case, and I think this the important point of this while thread (the choice of cor v. cov is case specific and needs to be determine by the person who understands the data & application best). Good luck with your research! $\endgroup$
    – Patrick
    Commented Jun 28, 2013 at 12:55
6
$\begingroup$

Speaking from a practical viewpoint - possibly unpopular here - if you have data measured on different scales, then go with correlation ('UV scaling' if you are a chemometrician), but if the variables are on the same scale and the size of them matters (e.g. with spectroscopic data), then covariance (centering the data only) makes more sense. PCA is a scale-dependent method and also log transformation can help with highly skewed data.

In my humble opinion based on 20 years of practical application of chemometrics you have to experiment a bit and see what works best for your type of data. At the end of the day you need to be able to reproduce your results and try to prove the predictability of your conclusions. How you get there is often a case of trial and error but the thing that matters is that what you do is documented and reproducible.

Improve this answer
$\endgroup$
1
  • 5
    $\begingroup$ The practical approach you seem to advocate here boils down to - when both covariances and correlations are warranted - "try both and see what works best". That pure empirical stance masks the fact that any choice goes with its own assumptions or paradigm about the reality which the researcher ought to be aware of in advance, even if he understands that he prefers one of them fully arbitrarily. Selecting "what works best" is the capitalizing on the feeling of pleasure, the narcomania. $\endgroup$
    – ttnphns
    Commented Jun 28, 2013 at 23:26
-1
$\begingroup$

I have no time to go into a fuller description of detailed & technical aspects of the experiment I described, and clarifications on wordings (recommending, performance, optimum) would again divert us away from the real issue, which is about what type of input data the PCA can(not) / should (not) be taking. PCA operates by taking linear combinations of numbers (values of variables). Mathematically, of course, one can add any two (real or complex) numbers. But if they have been re-scaled before PCA transformation, is their linear combination (and hence to process of maximization) still meaningful to operate on? If each variable $x_i$ has same variance $s^2$, then clearly yes, because $(x_1/s_1)+(x_2/s_2)=(x_1+x_2)/s$ is still proportional and comparable to the physical superposition of data $x_1+x_2$ itself. But if $s_1\not =s_2$, then the linear combination of standardized quantities distorts the data of the input variables to different degrees. There seems little point then to maximize the variance of their linear combination. In that case, PCA gives a solution for a different set of data, whereby each variable is scaled differently. If you then unstandardize afterwards (when using corr_PCA) then that may be OK and necessary; but if you just take the the raw corr_PCA solution as-is and stop there, you would obtain a mathematical solution, but not one related to the physical data. As unstandardization afterwards then seems mandatory as a minimum (i.e., 'unstretching' the axes by the inverse standard deviations), cov_PCA could have been used to begin with. If you are still reading by now, I am impressed! For now, I finish by quoting from Jolliffe's book, p. 42, which is the part that concerns me: 'It must not be forgotten, however, that correlation matrix PCs, when re-expressed in terms of the original variables, are still linear functions of x that maximize variance with respect to the standardized variables and not with respect to the original variables.' If you think I am interpreting this or its implications wrongly, this excerpt may be a good focus point for further discussion.

Improve this answer
$\endgroup$
14
  • 3
    $\begingroup$ It is so amusing that your own answer, which is in tune with everything people here were trying to convey to you, remains unsettled for you. You still argue There seems little point in PCA on correlations. Well, if you need to stay close to raw data ("physical data", as you strangely call it), you really shouldn't use correlations since they correspond to another ("distorted") data. $\endgroup$
    – ttnphns
    Commented Jun 28, 2013 at 20:42
  • 2
    $\begingroup$ (Cont.) Jolliffe's citation states, that PCs obtained on correlations will ever be themselves and cannot be turned "back" into PCs on covariances even though you can re-express them as linear combinations of the original variables. Thus, Jolliffe stresses the idea that PCA results are fully dependent on the type of pre-processing used and that there exist no "true", "genuine" or "universal" PCs... $\endgroup$
    – ttnphns
    Commented Jun 28, 2013 at 20:43
  • 2
    $\begingroup$ (Cont.) And in fact, Several lines below Jolliffe speaks of yet another "form" of PCA - PCA on X'X matrix. This form is even "closer" to original data than cov-PCA because no centering of variables are being done. And the results are usually utterly different. You could also do PCA on cosines. People do PCA on all versions of the SSCP matrix, albeit covariances or correlations are used most often. $\endgroup$
    – ttnphns
    Commented Jun 28, 2013 at 20:52
  • 5
    $\begingroup$ Underlying this answer is an implicit assumption that the units in which data are measured have an intrinsic meaning. That is rarely the case: we may choose to measure length in Angstroms, parsecs, or anything else, and time in picoseconds or millennia, without altering the meaning of the data one iota. The changes made in going from covariance to correlation are merely changes of units (which, by the way, are particularly sensitive to outlying data). This suggests the issue is not covariance versus correlation, but rather to find fruitful ways to express the data for analysis. $\endgroup$
    – whuber ♦
    Commented Jun 28, 2013 at 21:45
  • 3
    $\begingroup$ @ttnphns I'll stick by the "merely," thanks. Whether or not the implications are "profound," the fact remains that standardization of a variable literally is an affine re-expression of its values: a change in its units of measure. The importance of this observation lies in its implications for some claims appearing in this thread, of which the most prominent is "covariance-based PCA is the only truly correct one." Any conception of correctness that ultimately depends on an essentially arbitrary aspect of the data--how we write them down--cannot be right. $\endgroup$
    – whuber ♦
    Commented Jun 30, 2013 at 12:59

Not the answer you're looking for? Browse other questions tagged or ask your own question.