0
$\begingroup$

There are different ways of scaling variables in k-means clustering. I can divide all variable by sum of that column, I can take z score,I can have (max(var)-value)/(max(var)-min(var) etc.
How selection of variable scaling/normalising would change clustering result? I have done clustering on 1 million rows with 20 continuous variables. And the wss/tss is different for different scaling methods.

What is the reason from Algo perspective ?

$\endgroup$
  • $\begingroup$ @Anony -Mousse, yes, your answer of "drawbacks of K-means" is related to the my question, but at the same time how different scaling techniques would change the clustering results is what I am looking for. Actually, in my data science team only other people have got better clusters just by having different scaling techniques. So i wanted to know the reason. Also i am not very expert in the field so may be could't post question in right way but drawbacks related to scaling and different scaling effecting clustering results are indeed not exactly the same questions. :) $\endgroup$ – Arpit Sisodia Jan 29 '17 at 6:24
  • $\begingroup$ Sensitivity to scaling is a well-known drawback of k-means, so it should be included in the answers there. $\endgroup$ – Has QUIT--Anony-Mousse Jan 29 '17 at 8:41
  • $\begingroup$ How do you compare to your colleagues to decide what is better? I wouldn't be surprised if your solution becomes 'better' if you simply scale everything with 0.01 - indicating that the evaluation criteria is bad. $\endgroup$ – Has QUIT--Anony-Mousse Jan 29 '17 at 8:43
  • $\begingroup$ I have seen bss/tss to compare my and others results. They have got it better. $\endgroup$ – Arpit Sisodia Feb 4 '17 at 4:32
  • $\begingroup$ Sum of squares must not be compared across different normalization or data. As written above, scale your data set with a tiny constant close to zero to 'fake news" improve your scores. $\endgroup$ – Has QUIT--Anony-Mousse Feb 4 '17 at 18:49
1
$\begingroup$

Just generate a dataset

  • $y$ is always drawn from $N(0;100)$
  • $x$ is drawn from $N(-1;1)$ or $N(1;1)$

Two clusters, but k-means cannot find them because of it's sensitivity to scale. Because k-means is least-squares, it is particularly sensitive here. But in fact, no distance-based method will find this either, unless you scale the data, or use weights (e.g. a linear SVM can learn appropriate weights, and Mahalanobis distance computes weights based on covariance).

The underlying assumption of k-means that breaks with scaling is that a difference of $x$ in attribute 1 is exactly as much as a difference of $x$ in attribute 2 - and that a difference of $2x$ is actually $4$ times as severe (because it's a least squares approach!) If you scale your data differently, this can change substantially.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I agree with your point, and nice example to show it. Just a small nitpick about the 'no distance-based algorithm' part. A distance-based algorithm with the appropriate metric could work for your example (e.g. Mahalanobis distance with the right parameters). Hierarchical agglomerative clustering could also get it without rescaling, even using Euclidean distance. $\endgroup$ – user20160 Jan 28 '17 at 9:00
  • $\begingroup$ Mahalanobis is a form of weighting, so it belongs to the 'unless' clause. I will add it there. $\endgroup$ – Has QUIT--Anony-Mousse Jan 28 '17 at 9:16

Not the answer you're looking for? Browse other questions tagged or ask your own question.