0
$\begingroup$

I came across this post on Quora and the first answer cleared up why it might be a good idea to always normalize your data, but I want to understand what happens conversely. More specifically, what types of negative impacts would normalization of data have when using clustering?

$\endgroup$
6
  • 1
    $\begingroup$ We normalize so that a feature doesn't become artificially important just because it has a larger scale than the others. You might instead wind up amplifying a truly unimportant feature that initially has a very small range. But 'unimportant' is determined by the goals of the study. $\endgroup$ Mar 29, 2021 at 22:03
  • $\begingroup$ @AryaMcCarthy right, I think I understood that. But I want to know in what situations it might be a bad idea to normalize data in the context of $k$-means? EDIT: Sorry, by your second sentence, you mean if we normalize, we might give weight to a feature that may not be as important if we left our data not normalized? $\endgroup$
    – User_13
    Mar 29, 2021 at 22:05
  • $\begingroup$ That's exactly the context. If you have an unimportant feature that's now affecting how your data are clustered, your clusters are less meaningful than if you hadn't scaled the data. EDIT: Yep, that's what I mean. $\endgroup$ Mar 29, 2021 at 22:07
  • $\begingroup$ There's an example at stats.stackexchange.com/a/140723/919. $\endgroup$
    – whuber
    Mar 29, 2021 at 22:10
  • $\begingroup$ see also stats.stackexchange.com/q/372521/3277 $\endgroup$
    – ttnphns
    Mar 29, 2021 at 22:31

1 Answer 1

2
$\begingroup$

In some applications the measurement units of different variables are the same, and the meaning is actually pretty much the same. Normalisation is bad if in such a situation the variance is actually informative in the sense that a larger variance implies that the variable is more relevant. One such example that I have come across is voltage traces on different electrodes left by combustion of particles. An electrode with a low variance just isn't informative, it is dominated by noise, and the most relevant voltage signal on an electrode will correspond to the largest variance. If such data are normalised, the uninformative variables with the low variances will have their variances increased, so the noise will be amplified, whereas the more relevant variables will be relatively weighted down, which is undesirable and will make the clustering more dominated by noise. In this situation the different variables all correspond to the same kinds of measurements on the same scale, which suggests that normalisation is not necessarily required, however this doesn't suffice for it necessarily being bad. What makes it bad is the knowledge (or at least suspicion; these things are often not known for sure but rather suspected) that the clustering-relevant information is connected to a higher variance.

Another conceivable example are situations where some people are asked to rate different items (films, say) on a sufficiently detailed scale. If the aim is to cluster the users, the importance of the films to differentiate between them should be expected to be strongly connected to the variance; on a film with low variance opinions are just not that divided, and it would be a bad idea to make ratings on such films relatively more influential for clustering by normalisation.

More generally, abstract examples can be easily constructed in which large variance is caused by large distances between clusters, making such variables more informative for clustering. Consequently, normalising will be counterproductive in such cases as it will artificially decrease the between-cluster distances on these variables. This however is hard to diagnose unless the dimensionality is very low and things can be clearly seen on scatterplots, as it in principle requires the clusters to be already known, which is not normally the case.

But this illustrates that normalising is not always good; it is a simple (often fine but not always the best) remedy in cases in which the variables have wildly different variances and/or incompatible measurement scales, without any reason to believe that a larger variance may point to more information that a variable holds for clustering.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.