0
$\begingroup$

There are different ways of scaling variables in k-means clustering. I can divide all variable by sum of that column, I can take z score,I can have (max(var)-value)/(max(var)-min(var) etc.
How selection of variable scaling/normalising would change clustering result? I have done clustering on 1 million rows with 20 continuous variables. And the wss/tss is different for different scaling methods.

What is the reason from Algo perspective ?

Improve this question
$\endgroup$
6
  • $\begingroup$ @Anony -Mousse, yes, your answer of "drawbacks of K-means" is related to the my question, but at the same time how different scaling techniques would change the clustering results is what I am looking for. Actually, in my data science team only other people have got better clusters just by having different scaling techniques. So i wanted to know the reason. Also i am not very expert in the field so may be could't post question in right way but drawbacks related to scaling and different scaling effecting clustering results are indeed not exactly the same questions. :) $\endgroup$
    – Arpit Sisodia
    Commented Jan 29, 2017 at 6:24
  • $\begingroup$ Sensitivity to scaling is a well-known drawback of k-means, so it should be included in the answers there. $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Jan 29, 2017 at 8:41
  • $\begingroup$ How do you compare to your colleagues to decide what is better? I wouldn't be surprised if your solution becomes 'better' if you simply scale everything with 0.01 - indicating that the evaluation criteria is bad. $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Jan 29, 2017 at 8:43
  • $\begingroup$ I have seen bss/tss to compare my and others results. They have got it better. $\endgroup$
    – Arpit Sisodia
    Commented Feb 4, 2017 at 4:32
  • $\begingroup$ Sum of squares must not be compared across different normalization or data. As written above, scale your data set with a tiny constant close to zero to 'fake news" improve your scores. $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Feb 4, 2017 at 18:49

1 Answer 1

Reset to default
1
$\begingroup$

Just generate a dataset

  • $y$ is always drawn from $N(0;100)$
  • $x$ is drawn from $N(-1;1)$ or $N(1;1)$

Two clusters, but k-means cannot find them because of it's sensitivity to scale. Because k-means is least-squares, it is particularly sensitive here. But in fact, no distance-based method will find this either, unless you scale the data, or use weights (e.g. a linear SVM can learn appropriate weights, and Mahalanobis distance computes weights based on covariance).

The underlying assumption of k-means that breaks with scaling is that a difference of $x$ in attribute 1 is exactly as much as a difference of $x$ in attribute 2 - and that a difference of $2x$ is actually $4$ times as severe (because it's a least squares approach!) If you scale your data differently, this can change substantially.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I agree with your point, and nice example to show it. Just a small nitpick about the 'no distance-based algorithm' part. A distance-based algorithm with the appropriate metric could work for your example (e.g. Mahalanobis distance with the right parameters). Hierarchical agglomerative clustering could also get it without rescaling, even using Euclidean distance. $\endgroup$
    – user20160
    Commented Jan 28, 2017 at 9:00
  • $\begingroup$ Mahalanobis is a form of weighting, so it belongs to the 'unless' clause. I will add it there. $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Jan 28, 2017 at 9:16

Not the answer you're looking for? Browse other questions tagged or ask your own question.