PCA on Correlation or Covariance?

Question

What are the main differences between performing Principal Components Analysis on a correlation and covariance matrix? Do they give the same results?

Answer 1

You tend to use the covariance matrix when the variable scales are similar and the correlation matrix when variables are on different scales. Using the correlation matrix standardises the data.

In general they give different results. Especially when the scales are different.

As example, take a look a look at this R heptathlon data set. Some of the variables have an average value of about 1.8 (the high jump), whereas other variables (200m) are around 20.

library(HSAUR)
# look at heptathlon data
heptathlon

# correlations
round(cor(heptathlon[,-8]),2)   # ignoring "score" 
# covariance
round(cov(heptathlon[,-8]),2)

# PCA
# scale=T bases the PCA on the correlation matrix
hep.PC.cor = prcomp(heptathlon[,-8], scale=TRUE)
hep.PC.cov = prcomp(heptathlon[,-8], scale=FALSE)

# PC scores per competitor
hep.scores.cor = predict(hep.PC.cor)
hep.scores.cov = predict(hep.PC.cov)

# Plot of PC1 vs PC2
par(mfrow = c(2, 1))
plot(hep.scores.cov[,1],hep.scores.cov[,2],
     xlab="PC 1",ylab="PC 2", pch=NA, main="Covariance")
text(hep.scores.cov[,1],hep.scores.cov[,2],labels=1:25) 

plot(hep.scores.cor[,1],hep.scores.cor[,2],
     xlab="PC 1",ylab="PC 2", pch=NA, main="Correlation")
text(hep.scores.cor[,1],hep.scores.cor[,2],labels=1:25) 

Notice that the outlying individuals (in this data set) are outliers regardless of whether the covariance or correlation matrix is used.

Answer 2

Bernard Flury, in his excellent book introducing multivariate analysis, described this as an anti-property of principal components. It's actually worse than choosing between correlation or covariance. If you changed the units (e.g. US style gallons, inches etc. and EU style litres, centimetres) you will get substantively different projections of the data.

The argument against automatically using correlation matrices is that it is quite a brutal way of standardising your data. The problem with automatically using the covariance matrix, which is very apparent with that heptathalon data, is that the variables with the highest variance will dominate the first principal component (the variance maximising property).

So the "best" method to use is based on a subjective choice, careful thought and some experience.

Answer 3

A late reply, but you may find VERY useful handouts on multivariate data analysis "à la française" on the Bioinformatics department of Lyon. These come from the authors of the R ade4 package. It is in french, though.

Answer 4

If you have variables with widely varying scales, that is, caloric intake per day, gene expression, ELISA/Luminex in units of ug/dl, ng/dl, based on several orders of magnitude of protein expression, then use correlation as an input to PCA. However, if all of your data are based on e.g. gene expression from the same platform with similar range and scale, or you are working with log equity asset returns, then using correlation will throw out a tremendous amount of information. You actually don't need to think about the difference of using the correlation matrix R or covariance matrix C as an input to PCA, but rather, look at the diagonal values of C and R. You may observe a variance of 100 for one variable, and 10 on another -- which are on the diagonal of C. But when looking at the correlations, the diagonal contains all ones, so the variance of each variable is essentially changed to one as you use the R matrix. In summary, use the correlation matrix R when within-variable range and scale widely differs, and use C to preserve variance if the range and scale of variables is similar or in the same units of measure.