Reason to normalize in euclidean distance measures in hierarchical clustering

Question

Apparently, in hierarchical clustering in which the distance measure is Euclidean distance, the data must be first normalized or standardized to prevent the covariate with the highest variance from driving the clustering. Why is this? Isn't this fact desirable?

Answer 1

It depends on your data. And actually it has nothing to do with hierarchical clustering, but with the distance functions themselves.

The problem is when you have mixed attributes.

Say you have data on persons. Weight in grams and shoe size. Shoe sizes differ very little, while the differences in body mass (in grams) are much much larger. You can come up with dozens of examples. You just cannot compare 1 g and 1 shoe size difference. In fact, in this example you compute something that would have the physical unit of $\sqrt{g\cdot\text{shoe-size}}$!

Usually in these cases, Euclidean distance just does not make sense. But it may still work, in many situations if you normalize your data. Even if it actually doesn't make sense, it is a good heuristic for situations where you do not have "proven correct" distance function, such as Euclidean distance in human-scale physical world.

Answer 2

If you do not standardise your data then the variables measured in large valued units will dominate the computed dissimilarity and variables that are measured in small valued units will contribute very little.

We can visualise this in R via:

set.seed(42)
dat <- data.frame(var1 = rnorm(100, mean = 100000),
                  var2 = runif(100),
                  var3 = runif(100))
dist1 <- dist(dat)
dist2 <- dist(dat[,1, drop = FALSE])

dist1 contains the Euclidean distances for the 100 observations based on all three variables whilst dist2 contains the Euclidean distance based on var1 alone.

> summary(dist1)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
0.07351 0.77840 1.15200 1.36200 1.77000 5.30200 
> summary(dist2)
    Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
0.000072 0.470000 0.963600 1.169000 1.663000 5.280000

Note how similar the distributions of distances are, indicating little contribution from var2 and var3, and the actual distances are very similar:

> head(dist1)
[1] 1.9707186 1.0936524 0.8745579 1.2724471 1.6054603 0.1870085
> head(dist2)
[1] 1.9356566 1.0078300 0.7380958 0.9666901 1.4770830 0.1405636

If we standardise the data

dist3 <- dist(scale(dat))
dist4 <- dist(scale(dat[,1, drop = FALSE]))

then there is a big change in the distances based only on var1 and those based on all three variables:

> summary(dist3)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
0.09761 1.62400 2.25000 2.28200 2.93600 5.33100 
> summary(dist4)
    Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
0.000069 0.451400 0.925400 1.123000 1.597000 5.070000 
> head(dist3)
[1] 2.2636288 1.7272588 1.7791074 3.0129750 2.5821981 0.4434073
> head(dist4)
[1] 1.8587830 0.9678046 0.7087827 0.9282985 1.4184214 0.1349811

As hierarchical clustering uses these distances, whether it is desirable to standardise or not will depend on the type of data/variables you have and whether you want the big things to dominate the distances and hence dominant the formation of the clustering. The answer to this is domain specific and data-set specific.

Answer 3

Anony-Mousse gave an excellent answer. I would just add that the distance metric that makes sense would depend on the shape of the multivariate distributions. For multivariate Gaussian, the Mahalanobis distance is the appropriate measure.