19
$\begingroup$

I normalized my dataset then ran 3 component PCA to get small explained variance ratios ([0.50, 0.1, 0.05]).

When I didn't normalize but whitened my dataset then ran 3 component PCA, I got high explained variance ratios ([0.86, 0.06,0.01]).

Since I want to retain as much data into 3 components, should I NOT normalize the data? From my understanding we should always normalize before PCA.

By normalizing: setting mean to 0 and having unit variance.

$\endgroup$
  • 3
    $\begingroup$ Although it is unclear what you mean by "normalizing" the data (I know of at least four standard ways to do this in PCA and likely there are more), it sounds like the material at stats.stackexchange.com/questions/53 might be illuminating. $\endgroup$ – whuber Jul 2 '14 at 21:12
  • $\begingroup$ Hi whubber: I mean normalizing each observation to unit norm $\endgroup$ – user46925 Jul 2 '14 at 21:16
  • 1
    $\begingroup$ Thanks. The usual term for that is "standardizing." When you do that you are performing PCA based on correlations: that's why I think the link I provided might answer your question already. However, I see none of the answers there actually explain why or how you will get different results (perhaps because it's complex and the effect of standardization may be difficult to predict). $\endgroup$ – whuber Jul 2 '14 at 21:25
  • 1
    $\begingroup$ Is whitening before PCA typical? What's the goal of doing that? $\endgroup$ – shadowtalker Jul 3 '14 at 1:20
  • 2
    $\begingroup$ If you were working with images, for example, the norm of the images corresponds to the brightness. The high explained variance of the un-normalized data means that a lot of the data can be explained by changes in brightness. If brightness is not important to you, as it often is not in image processing, then you would want to make all the images unit norm first. Even thought the explained variance of your pca components will be lower, it better reflects what you're interested in. $\endgroup$ – Aaron Jul 3 '14 at 14:25
37
$\begingroup$

Depends on the goal of your analysis. Some common practices, some of which are mentioned in whuber's link:

  1. Standardizing is usually done when the variables on which the PCA is performed are not measured on the same scale. Note that standardizing implies assigning equal importance to all variables.
  2. If they are not measured on the same scale and you choose to work on the non standardized variables, it is often the case that each PC is dominated by a single variable and you just get a sort of ordering of the variables by their variance. (One of the loadings of each (early) component will be close to +1 or -1.)
  3. The two methods often lead to different results, as you have experienced.

Intuitive example:

Suppose you have two variables: the height of a tree and the girth of the same tree. We will convert the volume to a factor: a tree will be high in volume if its volume is bigger than 20 cubic feet, and low in volume otherwise. We will use the trees dataset which comes preloaded in R.

>data(trees)
>tree.girth<-trees[,1]
>tree.height<-trees[,2]
>tree.vol<-as.factor(ifelse(trees[,3]>20,"high","low"))

Now suppose that the height was actually measured in miles instead of feet.

>tree.height<-tree.height/5280
>tree<-cbind(tree.height,tree.girth)
>
>#do the PCA
>tree.pca<-princomp(tree)
>summary(tree.pca)
Importance of components:
                      Comp.1       Comp.2
Standard deviation     3.0871086 1.014551e-03
Proportion of Variance 0.9999999 1.080050e-07
Cumulative Proportion  0.9999999 1.000000e+00

The first component explains almost 100% of the variability in the data. The loadings:

> loadings(tree.pca)

Loadings:
            Comp.1 Comp.2
tree.height        -1    
tree.girth   1           

Graphical assessment:

>biplot(tree.pca,xlabs=tree.vol,col=c("grey","red"))

Biplot of first method

We see that trees high in volume tend to have a high tree girth, but the three height doesn't give any information on tree volume. This is likely wrong and the consequence of the two different unit measures.

We could use the same units, or we could standardize the variables. I expect both will lead to a more balanced picture of the variability. Of course in this case one can argue that the variables should have the same unit but not be standardized, which may be a valid argument, were it not that we are measuring two different things. (When we would be measuring the weight of the tree and the girth of the tree, the scale on which both should be measured is no longer very clear. In this case we have a clear argument to work on the standardized variables.)

>tree.height<-tree.height*5280
>tree<-cbind(tree.height,tree.girth)
>
>#do the PCA
>tree.pca<-princomp(tree)
> summary(tree.pca)
Importance of components:
                          Comp.1    Comp.2
Standard deviation     6.5088696 2.5407042
Proportion of Variance 0.8677775 0.1322225
Cumulative Proportion  0.8677775 1.0000000
> loadings(tree.pca)

Loadings:
            Comp.1 Comp.2
tree.height -0.956  0.293
tree.girth  -0.293 -0.956

>biplot(tree.pca,xlabs=tree.vol,col=c("grey","red"))

Biplot of second method

We now see that trees which are tall and have a big girth, are high in volume (bottom left corner), compared to low girth and low height for low volume trees (upper right corner). This intuitively makes sense.

If one watches closely, however, we see that the contrast between high/low volume is strongest in the girth direction and not in the height direction. Let's see what happens when we standardize:

>tree<-scale(tree,center=F,scale=T)
>tree.pca<-princomp(tree)
> summary(tree.pca)
Importance of components:
                          Comp.1     Comp.2
Standard deviation     0.2275561 0.06779544
Proportion of Variance 0.9184749 0.08152510
Cumulative Proportion  0.9184749 1.00000000
> loadings(tree.pca)

Loadings:
            Comp.1 Comp.2
tree.height  0.203 -0.979
tree.girth   0.979  0.203
>biplot(tree.pca,xlabs=tree.vol,col=c("grey","red"))

Biplot of third method

Indeed, the girth now explains the majority of the difference in high and low volume trees! (The length of the arrow in the biplot is indicatory of the variance in the original variable.) So even if things are measured on the same scale, standardizing may be useful. Not standardizing may be recommended when we are for example comparing the length of different species of trees because this is exactly the same measurement.

$\endgroup$
  • 2
    $\begingroup$ (+1) Very nice answer, actually surpassing all those given in the linked popular thread about PCA on covariance vs. correlation. $\endgroup$ – amoeba says Reinstate Monica Jul 3 '14 at 10:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy