PCA on correlation or covariance?

Question

What are the main differences between performing principal component analysis (PCA) on the correlation matrix and on the covariance matrix? Do they give the same results?

Answer 1

You tend to use the covariance matrix when the variable scales are similar and the correlation matrix when variables are on different scales.

Using the correlation matrix is equivalent to standardizing each of the variables (to mean 0 and standard deviation 1). In general, PCA with and without standardizing will give different results. Especially when the scales are different.

As an example, take a look at this R heptathlon data set. Some of the variables have an average value of about 1.8 (the high jump), whereas other variables (run 800m) are around 120.

library(HSAUR)
heptathlon[,-8]      # look at heptathlon data (excluding 'score' variable)

This outputs:

                   hurdles highjump  shot run200m longjump javelin run800m
Joyner-Kersee (USA)   12.69     1.86 15.80   22.56     7.27   45.66  128.51
John (GDR)            12.85     1.80 16.23   23.65     6.71   42.56  126.12
Behmer (GDR)          13.20     1.83 14.20   23.10     6.68   44.54  124.20
Sablovskaite (URS)    13.61     1.80 15.23   23.92     6.25   42.78  132.24
Choubenkova (URS)     13.51     1.74 14.76   23.93     6.32   47.46  127.90
...

Now let's do PCA on covariance and on correlation:

# scale=T bases the PCA on the correlation matrix
hep.PC.cor = prcomp(heptathlon[,-8], scale=TRUE)
hep.PC.cov = prcomp(heptathlon[,-8], scale=FALSE)

biplot(hep.PC.cov)
biplot(hep.PC.cor)  

Notice that PCA on covariance is dominated by run800m and javelin: PC1 is almost equal to run800m (and explains $82\%$ of the variance) and PC2 is almost equal to javelin (together they explain $97\%$). PCA on correlation is much more informative and reveals some structure in the data and relationships between variables (but note that the explained variances drop to $64\%$ and $71\%$).

Notice also that the outlying individuals (in this data set) are outliers regardless of whether the covariance or correlation matrix is used.

Answer 2

Bernard Flury, in his excellent book introducing multivariate analysis, described this as an anti-property of principal components. It's actually worse than choosing between correlation or covariance. If you changed the units (e.g. US style gallons, inches etc. and EU style litres, centimetres) you will get substantively different projections of the data.

The argument against automatically using correlation matrices is that it is quite a brutal way of standardising your data. The problem with automatically using the covariance matrix, which is very apparent with that heptathalon data, is that the variables with the highest variance will dominate the first principal component (the variance maximising property).

So the "best" method to use is based on a subjective choice, careful thought and some experience.

Answer 3

UNTRANSFORMED (RAW) DATA: If you have variables with widely varying scales for raw, untransformed data, that is, caloric intake per day, gene expression, ELISA/Luminex in units of ug/dl, ng/dl, based on several orders of magnitude of protein expression, then use correlation as an input to PCA. However, if all of your data are based on e.g. gene expression from the same platform with similar range and scale, or you are working with log equity asset returns, then using correlation will throw out a tremendous amount of information.

You actually don't need to think about the difference of using the correlation matrix $\mathbf{R}$ or covariance matrix $\mathbf{C}$ as an input to PCA, but rather, look at the diagonal values of $\mathbf{C}$ and $\mathbf{R}$. You may observe a variance of $100$ for one variable, and $10$ on another -- which are on the diagonal of $\mathbf{C}$. But when looking at the correlations, the diagonal contains all ones, so the variance of each variable is essentially changed to $1$ as you use the $\mathbf{R}$ matrix.

TRANSFORMED DATA: If the data have been transformed via normalization, percentiles, or mean-zero standardization (i.e., $Z$-scores), so that the range and scale of all the continuous variables is the same, then you could use the Covariance matrix $\mathbf{C}$ without any problems. (correlation will mean-zero standardize variables). Recall, however, that these transformations will not remove skewness (i.e., left or right tails in histograms) in your variables prior to running PCA. Typical PCA analysis does not involve removal of skewness; however, some readers may need to remove skewness to meet strict normality constraints.

In summary, use the correlation matrix $\mathbf{R}$ when within-variable range and scale widely differs, and use the covariance matrix $\mathbf{C}$ to preserve variance if the range and scale of variables is similar or in the same units of measure.

SKEWED VARIABLES: If any of the variables are skewed with left or right tails in their histograms, i.e., the Shapiro-Wilk or Lilliefors normality test is significant $(P<0.05)$, then there may be some issues if you need to apply the normality assumption. In this case, use the van der Waerden scores (transforms) determined from each variable. The van der Waerden (VDW) score for a single observation is merely the inverse cumulative (standard) normal mapping of the observation's percentile value. For example, say you have $n=100$ observations for a continuous variable, you can determine the VDW scores using:

1. First, sort the values in ascending order, then assign ranks, so you would obtain ranks of $R_i=1,2,\ldots,100.$
2. Next, determine the percentile for each observation as $pct_i=R_i/(n+1)$.
3. Once the percentile values are obtained, input them into the inverse mapping function for the CDF of the standard normal distribution, i.e., $N(0,1)$, to obtain the $Z$-score for each, using $Z_i=\Phi^{-1}(pct_i)$.

For example, if you plug in a $pct_i$ value 0.025, you will get $-1.96=\Phi^{-1}(0.025)$. Same goes for a plugin value of $pct_i=0.975$, you'll get $1.96=\Phi^{-1}(0.975)$.

Use of VDW scores is very popular in genetics, where many variables are transformed into VDW scores, and then input into analyses. The advantage of using VDW scores is that skewness and outlier effects are removed from the data, and can be used if the goal is to perform an analysis under the contraints of normality -- and every variable needs to be purely standard normal distributed with no skewness or outliers.

Answer 4

A common answer is to suggest that covariance is used when variables are on the same scale, and correlation when their scales are different. However, this is only true when scale of the variables isn't a factor. Otherwise, why would anyone ever do covariance PCA? It would be safer to always perform correlation PCA.

Imagine that your variables have different units of measure, such as meters and kilograms. It shouldn't matter whether you use meters or centimeters in this case, so you could argue that correlation matrix should be used.

Consider now population of people in different states. The units of measure are the same - counts (number) of people. Now, the scales could be different: Vermont has 600K and CA - 38M people. Should we use correlation matrix here? It depends. In some applications we do want to adjust for the size of the state. Using the covariance matrix is one way for building factors that account for the size of the state.

Hence, my answer is to use covariance matrix when variance of the original variable is important, and use correlation when it is not.

Answer 5

I personally find it very valuable to discuss these options in light of the maximum-likelihood principal component analysis model (MLPCA) [1,2]. In MLPCA one applies a scaling (or even a rotation) such that the measurement errors in the measured variables are independent and distributed according to the standard normal distribution. This scaling is also known as maximum likelihood scaling (MALS) [3]. In some case, the PCA model and the parameter defining the MALS scaling/rotation can be estimated together [4].

To interpret correlation-based and covariance-based PCA, one can then argue that:

1. Covariance-based PCA is equivalent to MLPCA whenever the variance-covariance matrix of the measurement errors is assumed diagonal with equal elements on its diagonal. The measurement error variance parameter can then be estimated by applying the probabilistic principal component analysis (PPCA) model [5]. I find this a reasonable assumption in several cases I have studied, specifically when all measurements are of the same type of variable (e.g. all flows, all temperatures, all concentrations, or all absorbance measurements). Indeed, it can be safe to assume that the measurement errors for such variables are distributed independently and identically.
2. Correlation-based PCA is equivalent to MLPCA whenever the variance-covariance matrix of the measurement errors is assumed diagonal with each element on the diagonal proportional to the overall variance of the corresponding measured variable. While this is a popular method, I personally find the proportionality assumption unreasonable in most cases I study. As a consequence, this means I cannot interpret correlation-based PCA as an MLPCA model. In the cases where (1) the implied assumptions of covariance-based PCA do not apply and (2) an MLPCA interpretation is valuable, I recommend to use one of the MLPCA methods instead [1-4].
3. Correlation-based and covariance-based PCA will produce the exact same results -apart from a scalar multiplier- when the individual variances for each variable are all exactly equal to each other. When these individual variances are similar but not the same, both methods will produce similar results.

As stressed above already, the ultimate choice depends on the assumptions you are making. In addition, the utility of any particular model depends also on the context and purpose of your analysis. To quote George E. P. Box: "All models are wrong, but some are useful".

[1] Wentzell, P. D., Andrews, D. T., Hamilton, D. C., Faber, K., & Kowalski, B. R. (1997). Maximum likelihood principal component analysis. Journal of Chemometrics, 11(4), 339-366.

[2] Wentzell, P. D., & Lohnes, M. T. (1999). Maximum likelihood principal component analysis with correlated measurement errors: theoretical and practical considerations. Chemometrics and Intelligent Laboratory Systems, 45(1-2), 65-85.

[3] Hoefsloot, H. C., Verouden, M. P., Westerhuis, J. A., & Smilde, A. K. (2006). Maximum likelihood scaling (MALS). Journal of Chemometrics, 20(3‐4), 120-127.

[4] Narasimhan, S., & Shah, S. L. (2008). Model identification and error covariance matrix estimation from noisy data using PCA. Control Engineering Practice, 16(1), 146-155.

[5] Tipping, M. E., & Bishop, C. M. (1999). Probabilistic principal component analysis. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 61(3), 611-622.

Answer 6

Straight and simple: if the scales are similar use cov-PCA, if not, use corr-PCA; otherwise, you better have a defense for not. If in doubt, use an F-test for the equality of the variances (ANOVA). If it fails the F-test, use corr; otherwise, use cov.

Answer 7

The arguments based on scale (for variables expressed in the same physical units) seem rather weak. Imagine a set of (dimensionless) variables whose standard deviations vary between 0.001 and 0.1. Compared to a standardized value of 1, these both seem to be 'small' and comparable levels of fluctuations. However, when you express them in decibel, this gives a range of -60 dB against -10 and 0 dB, respectively. Then this would probably then be classified as a 'large range' -- especially if you would include a standard deviation close to 0, i.e., minus infinity dB.

My suggestion would be to do BOTH a correlation- and covariance-based PCA. If the two give the same (or very similar, whatever this may mean) PCs, then you can be reassured you've got an answer that is meaningul. If they give widely different PCs don't use PCA, because two different answers to one problem is not sensible way to solve questions.