Not normalizing data before PCA gives better explained variance ratio

Question

I normalized my dataset then ran 3 component PCA to get small explained variance ratios ([0.50, 0.1, 0.05]).

When I didn't normalize but whitened my dataset then ran 3 component PCA, I got high explained variance ratios ([0.86, 0.06,0.01]).

Since I want to retain as much data into 3 components, should I NOT normalize the data? From my understanding we should always normalize before PCA.

By normalizing: setting mean to 0 and having unit variance.

Answer 1

Depends on the goal of your analysis. Some common practices, some of which are mentioned in whuber's link:

1. Standardizing is usually done when the variables on which the PCA is performed are not measured on the same scale. Note that standardizing implies assigning equal importance to all variables.
2. If they are not measured on the same scale and you choose to work on the non standardized variables, it is often the case that each PC is dominated by a single variable and you just get a sort of ordering of the variables by their variance. (One of the loadings of each (early) component will be close to +1 or -1.)
3. The two methods often lead to different results, as you have experienced.

Intuitive example:

Suppose you have two variables: the height of a tree and the girth of the same tree. We will convert the volume to a factor: a tree will be high in volume if its volume is bigger than 20 cubic feet, and low in volume otherwise. We will use the trees dataset which comes preloaded in R.

>data(trees)
>tree.girth<-trees[,1]
>tree.height<-trees[,2]
>tree.vol<-as.factor(ifelse(trees[,3]>20,"high","low"))

Now suppose that the height was actually measured in miles instead of feet.

>tree.height<-tree.height/5280
>tree<-cbind(tree.height,tree.girth)
>
>#do the PCA
>tree.pca<-princomp(tree)
>summary(tree.pca)
Importance of components:
                      Comp.1       Comp.2
Standard deviation     3.0871086 1.014551e-03
Proportion of Variance 0.9999999 1.080050e-07
Cumulative Proportion  0.9999999 1.000000e+00

The first component explains almost 100% of the variability in the data. The loadings:

> loadings(tree.pca)

Loadings:
            Comp.1 Comp.2
tree.height        -1    
tree.girth   1           

Graphical assessment:

>biplot(tree.pca,xlabs=tree.vol,col=c("grey","red"))

We see that trees high in volume tend to have a high tree girth, but the three height doesn't give any information on tree volume. This is likely wrong and the consequence of the two different unit measures.

We could use the same units, or we could standardize the variables. I expect both will lead to a more balanced picture of the variability. Of course in this case one can argue that the variables should have the same unit but not be standardized, which may be a valid argument, were it not that we are measuring two different things. (When we would be measuring the weight of the tree and the girth of the tree, the scale on which both should be measured is no longer very clear. In this case we have a clear argument to work on the standardized variables.)

>tree.height<-tree.height*5280
>tree<-cbind(tree.height,tree.girth)
>
>#do the PCA
>tree.pca<-princomp(tree)
> summary(tree.pca)
Importance of components:
                          Comp.1    Comp.2
Standard deviation     6.5088696 2.5407042
Proportion of Variance 0.8677775 0.1322225
Cumulative Proportion  0.8677775 1.0000000
> loadings(tree.pca)

Loadings:
            Comp.1 Comp.2
tree.height -0.956  0.293
tree.girth  -0.293 -0.956

>biplot(tree.pca,xlabs=tree.vol,col=c("grey","red"))

We now see that trees which are tall and have a big girth, are high in volume (bottom left corner), compared to low girth and low height for low volume trees (upper right corner). This intuitively makes sense.