154
$\begingroup$

What are the main differences between performing principal component analysis (PCA) on the correlation matrix and on the covariance matrix? Do they give the same results?

$\endgroup$
131
$\begingroup$

You tend to use the covariance matrix when the variable scales are similar and the correlation matrix when variables are on different scales.

Using the correlation matrix is equivalent to standardizing each of the variables (to mean 0 and standard deviation 1). In general, PCA with and without standardizing will give different results. Especially when the scales are different.

As an example, take a look at this R heptathlon data set. Some of the variables have an average value of about 1.8 (the high jump), whereas other variables (run 800m) are around 120.

library(HSAUR)
heptathlon[,-8]      # look at heptathlon data (excluding 'score' variable)

This outputs:

                   hurdles highjump  shot run200m longjump javelin run800m
Joyner-Kersee (USA)   12.69     1.86 15.80   22.56     7.27   45.66  128.51
John (GDR)            12.85     1.80 16.23   23.65     6.71   42.56  126.12
Behmer (GDR)          13.20     1.83 14.20   23.10     6.68   44.54  124.20
Sablovskaite (URS)    13.61     1.80 15.23   23.92     6.25   42.78  132.24
Choubenkova (URS)     13.51     1.74 14.76   23.93     6.32   47.46  127.90
...

Now let's do PCA on covariance and on correlation:

# scale=T bases the PCA on the correlation matrix
hep.PC.cor = prcomp(heptathlon[,-8], scale=TRUE)
hep.PC.cov = prcomp(heptathlon[,-8], scale=FALSE)

biplot(hep.PC.cov)
biplot(hep.PC.cor)  

PCA on correlation or covariance

Notice that PCA on covariance is dominated by run800m and javelin: PC1 is almost equal to run800m (and explains $82\%$ of the variance) and PC2 is almost equal to javelin (together they explain $97\%$). PCA on correlation is much more informative and reveals some structure in the data and relationships between variables (but note that the explained variances drop to $64\%$ and $71\%$).

Notice also that the outlying individuals (in this data set) are outliers regardless of whether the covariance or correlation matrix is used.

$\endgroup$
  • $\begingroup$ What is the situation, if I convert the variables to z-scores first? $\endgroup$ – Jirka-x1 May 18 '13 at 16:00
  • 9
    $\begingroup$ @Jirka-x1 the covariance matrix of standardized variables (i.e. z scores) equals the correlation matrix. $\endgroup$ – Alexis Jul 2 '14 at 21:22
  • $\begingroup$ @Alexis Can it therefore be inferred that the covariance matrix of standardised variables equals the correlation matrix of standardised variables? $\endgroup$ – j b Sep 1 '17 at 10:13
  • 1
    $\begingroup$ @JamieBullock $\mathbf{\Sigma}$ (covariance matrix) for standardized data = $\mathbf{R}$ (correlation matrix). $\mathbf{R} = \mathbf{R}$ whether or not the data are standardized (correlation is insensitive to linear transformations of the data. So , for example, if you have $X$ and $Y$ and they correlate with $r_{XY}$, then if $X^{*} = aX+b$ and $Y^{*} = aY+b$ $X^{*}$ and $Y^{*}$ also correlate with $r_{XY}$). $\endgroup$ – Alexis Sep 1 '17 at 23:23
  • $\begingroup$ One important notice: when using covariance in your PCA, your PCs will not be correlated among each other, which does not hold true for correlation-based PCA. This is especially important when intending to perform PCA prior to regression in a multicolinear set of explanatory variables. However the theory behind this is not clear. Could anyone shed some light on this difference? $\endgroup$ – ouranos Sep 25 '18 at 12:41
54
$\begingroup$

Bernard Flury, in his excellent book introducing multivariate analysis, described this as an anti-property of principal components. It's actually worse than choosing between correlation or covariance. If you changed the units (e.g. US style gallons, inches etc. and EU style litres, centimetres) you will get substantively different projections of the data.

The argument against automatically using correlation matrices is that it is quite a brutal way of standardising your data. The problem with automatically using the covariance matrix, which is very apparent with that heptathalon data, is that the variables with the highest variance will dominate the first principal component (the variance maximising property).

So the "best" method to use is based on a subjective choice, careful thought and some experience.

$\endgroup$
46
$\begingroup$

UNTRANSFORMED (RAW) DATA: If you have variables with widely varying scales for raw, untransformed data, that is, caloric intake per day, gene expression, ELISA/Luminex in units of ug/dl, ng/dl, based on several orders of magnitude of protein expression, then use correlation as an input to PCA. However, if all of your data are based on e.g. gene expression from the same platform with similar range and scale, or you are working with log equity asset returns, then using correlation will throw out a tremendous amount of information.

You actually don't need to think about the difference of using the correlation matrix $\mathbf{R}$ or covariance matrix $\mathbf{C}$ as an input to PCA, but rather, look at the diagonal values of $\mathbf{C}$ and $\mathbf{R}$. You may observe a variance of $100$ for one variable, and $10$ on another -- which are on the diagonal of $\mathbf{C}$. But when looking at the correlations, the diagonal contains all ones, so the variance of each variable is essentially changed to $1$ as you use the $\mathbf{R}$ matrix.

TRANSFORMED DATA: If the data have been transformed via normalization, percentiles, or mean-zero standardization (i.e., $Z$-scores), so that the range and scale of all the continuous variables is the same, then you could use the Covariance matrix $\mathbf{C}$ without any problems. (correlation will mean-zero standardize variables). Recall, however, that these transformations will not remove skewness (i.e., left or right tails in histograms) in your variables prior to running PCA. Typical PCA analysis does not involve removal of skewness; however, some readers may need to remove skewness to meet strict normality constraints.

In summary, use the correlation matrix $\mathbf{R}$ when within-variable range and scale widely differs, and use the covariance matrix $\mathbf{C}$ to preserve variance if the range and scale of variables is similar or in the same units of measure.

SKEWED VARIABLES: If any of the variables are skewed with left or right tails in their histograms, i.e., the Shapiro-Wilk or Lilliefors normality test is significant $(P<0.05)$, then there may be some issues if you need to apply the normality assumption. In this case, use the van der Waerden scores (transforms) determined from each variable. The van der Waerden (VDW) score for a single observation is merely the inverse cumulative (standard) normal mapping of the observation's percentile value. For example, say you have $n=100$ observations for a continuous variable, you can determine the VDW scores using:

  1. First, sort the values in ascending order, then assign ranks, so you would obtain ranks of $R_i=1,2,\ldots,100.$
  2. Next, determine the percentile for each observations as $pct_i=R_i/(n+1)$.
  3. Once the percentile values are obtained, input them into the inverse mapping function for the CDF of the standard normal distribution, i.e., $N(0,1)$, to obtain the $Z$-score for each, using $Z_i=\Phi^{-1}(pct_i)$.

For example, if you plug in a $pct_i$ value 0.025, you will get $-1.96=\Phi^{-1}(0.025)$. Same goes for a plugin value of $pct_i=0.975$, you'll get $1.96=\Phi^{-1}(0.975)$.

Use of VDW scores is very popular in genetics, where many variables are transformed into VDW scores, and then input into analyses. The advantage of using VDW scores is that skewness and outlier effects are removed from the data, and can be used if the goal is to perform an analysis under the contraints of normality -- and every variable needs to be purely standard normal distributed with no skewness or outliers.

$\endgroup$
  • 7
    $\begingroup$ This is by far the most sensible answer here, as it actually gives a proper view that covariance wins when appropriate. Too many answers here and elsewhere mention the usual "it depends" without actually giving a hard basis for why one should prefer covariance if possible. Here lep does: covariance doesn't chuck out any of the info which correlation does. The stock data example is a good one: high beta stocks will of course have higher loadings but they probably should, just like any facet of any analysis that is more volatile is usually more interesting (within reason). $\endgroup$ – Thomas Browne Nov 12 '14 at 15:27
  • 3
    $\begingroup$ Of course the problem at hand must be analysed for whether or not higher variance is an interesting facet of the analysis. If it is not, then of course correl is better, and that definitely holds if units are different. $\endgroup$ – Thomas Browne Nov 12 '14 at 15:30
  • $\begingroup$ Great answer +1. I guess one more example might be applying PCA to term structure analysis on bond yields in finance. Variances of yields on varied maturities vary, but since they are all yields, the varying scales are normally not unacceptably wide. Indeed, more/less volatility of certain maturity yield itself provides rich information. $\endgroup$ – Nicholas Nov 11 '16 at 17:51
11
+100
$\begingroup$

A common answer is to suggest that covariance is used when variables are on the same scale, and correlation when their scales are different. However, this is only true when scale of the variables isn't a factor. Otherwise, why would anyone ever do covariance PCA? It would be safer to always perform correlation PCA.

Imagine that your variables have different units of measure, such as meters and kilograms. It shouldn't matter whether you use meters or centimeters in this case, so you could argue that correlation matrix should be used.

Consider now population of people in different states. The units of measure are the same - counts (number) of people. Now, the scales could be different: DC has 600K and CA - 38M people. Should we use correlation matrix here? It depends. In some applications we do want to adjust for the size of the state. Using the covariance matrix is one way for building factors that account for the size of the state.

Hence, my answer is to use covariance matrix when variance of the original variable is important, and use correlation when it is not.

$\endgroup$
2
$\begingroup$

I personally find it very valuable to discuss these options in light of the maximum-likelihood principal component analysis model (MLPCA) [1,2]. In MLPCA one applies a scaling (or even a rotation) such that the measurement errors in the measured variables are independent and distributed according to the standard normal distribution. This scaling is also known as maximum likelihood scaling (MALS) [3]. In some case, the PCA model and the parameter defining the MALS scaling/rotation can be estimated together [4].

To interpret correlation-based and covariance-based PCA, one can then argue that:

  1. Covariance-based PCA is equivalent to MLPCA whenever the variance-covariance matrix of the measurement errors is assumed diagonal with equal elements on its diagonal. The measurement error variance parameter can then be estimated by applying the probabilistic principal component analysis (PPCA) model [5]. I find this a reasonable assumption in several cases I have studied, specifically when all measurements are of the same type of variable (e.g. all flows, all temperatures, all concentrations, or all absorbance measurements). Indeed, it can be safe to assume that the measurement errors for such variables are distributed independently and identically.
  2. Correlation-based PCA is equivalent to MLPCA whenever the variance-covariance matrix of the measurement errors is assumed diagonal with each element on the diagonal proportional to the overall variance of the corresponding measured variable. While this is a popular method, I personally find the proportionality assumption unreasonable in most cases I study. As a consequence, this means I cannot interpret correlation-based PCA as an MLPCA model. In the cases where (1) the implied assumptions of covariance-based PCA do not apply and (2) an MLPCA interpretation is valuable, I recommend to use one of the MLPCA methods instead [1-4].
  3. Correlation-based and covariance-based PCA will produce the exact same results -apart from a scalar multiplier- when the individual variances for each variable are all exactly equal to each other. When these individual variances are similar but not the same, both methods will produce similar results.

As stressed above already, the ultimate choice depends on the assumptions you are making. In addition, the utility of any particular model depends also on the context and purpose of your analysis. To quote George E. P. Box: "All models are wrong, but some are useful".

[1] Wentzell, P. D., Andrews, D. T., Hamilton, D. C., Faber, K., & Kowalski, B. R. (1997). Maximum likelihood principal component analysis. Journal of Chemometrics, 11(4), 339-366.

[2] Wentzell, P. D., & Lohnes, M. T. (1999). Maximum likelihood principal component analysis with correlated measurement errors: theoretical and practical considerations. Chemometrics and Intelligent Laboratory Systems, 45(1-2), 65-85.

[3] Hoefsloot, H. C., Verouden, M. P., Westerhuis, J. A., & Smilde, A. K. (2006). Maximum likelihood scaling (MALS). Journal of Chemometrics, 20(3‐4), 120-127.

[4] Narasimhan, S., & Shah, S. L. (2008). Model identification and error covariance matrix estimation from noisy data using PCA. Control Engineering Practice, 16(1), 146-155.

[5] Tipping, M. E., & Bishop, C. M. (1999). Probabilistic principal component analysis. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 61(3), 611-622.

$\endgroup$
-1
$\begingroup$

Straight and simple: if the scales are similar use cov-PCA, if not, use corr-PCA; otherwise, you better have a defense for not. If in doubt, use an F-test for the equality of the variances (ANOVA). If it fails the F-test, use corr; otherwise, use cov.

$\endgroup$
  • 2
    $\begingroup$ -1. I don't see why running an F-test could be relevant here. PCA is an exploratory method, not a confirmatory one (as statistical tests are). $\endgroup$ – amoeba Dec 18 '17 at 9:14
-5
$\begingroup$

The arguments based on scale (for variables expressed in the same physical units) seem rather weak. Imagine a set of (dimensionless) variables whose standard deviations vary between 0.001 and 0.1. Compared to a standardized value of 1, these both seem to be 'small' and comparable levels of fluctuations. However, when you express them in decibel, this gives a range of -60 dB against -10 and 0 dB, respectively. Then this would probably then be classified as a 'large range' -- especially if you would include a standard deviation close to 0, i.e., minus infinity dB.

My suggestion would be to do BOTH a correlation- and covariance-based PCA. If the two give the same (or very similar, whatever this may mean) PCs, then you can be reassured you've got an answer that is meaningul. If they give widely different PCs don't use PCA, because two different answers to one problem is not sensible way to solve questions.

$\endgroup$
  • 9
    $\begingroup$ (-1) Getting "two different answers to the same problem" often just means you're bashing away mindlessly without thinking about which technique is appropriate for your analytical aims. It does not mean that one or (as you state) both techniques are not sensible, but only that at least one might not be appropriate for the problem or the data. Furthermore, in many cases you can anticipate that covariance-based PCA and correlation-based PCA should give different answers. After all, they are measuring different aspects of the data. Doing both by default would not make sense. $\endgroup$ – whuber Jun 27 '13 at 15:00
  • $\begingroup$ Actually it is very reasonable to get 2 different answers when using PCA with correlation and covariance. In the stock case, it is a question of whether you should take betas (or standard deviation) into account $\endgroup$ – Juancentro Nov 22 '14 at 17:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.