How to compare clustering results between "raw" and normalized data I have a dataset and I would like to apply a clustering algorithm to find some groups. I do not have any label, so it is just wondering if I can find relevant clusters. If it may help, it is information about some players (say age, height, weight, and some other performance metrics - the point is there are features with different scalings...). Also, although I think my question applies to any clustering algorithm, I am currently using K-Means.
Now, I just applied a K-Means with 5 clusters on my data and I calculated some very standard performance metrics: inertia and silhouette score (overall average, and average per cluster).
Then, given that scaling data is considered a best practice, I decided to standardize every feature (z-score) and apply again K-Means with 5 clusters. I computed the same metrics. I expected inertia to be lower (it is a sum of squared distances...) but surprisingly all the silhouette scores were lower, too! I expected silhouette to improve on scaled data. ¿Why wasn't that the case?
This led me to ask some questions: ¿is it enough to conclude that in this case it is better not to scale the data or am I missing something? Can we actually compare these two silhouette scores, despite being computed on data with different scalings? If not, which would be a better approach to evaluation / comparison?
Thank you in advance for any answer / suggestion / thought.
Cheers!
 A: There are different considerations here. The Euclidean distances between points are different with and without scaling. The "optimal" choice of distances depends on the meaning of the data and what should qualify data to be in the same cluster. Scaling will basically unify the influence of the different variables on the clustering, which is very often a reasonable thing to do, particularly if the variables have different scales, the values of which cannot directly be compared. If I understand your data correctly, scaling looks like a sensible thing to do, although in general it also depends on the aim and potential use of the resulting clusters. Note also that there are various ways of standardising the data, instead of z-scores one can also use median and MAD scaling or scaling to 0-1 range, which in particular treat outliers differently (there is no clear right and wrong here unfortunately, with or without outliers all these have pros and cons).
Regarding inertia (I guess you mean what I'd call within-cluster mean squares), inertia values cannot be compared between different distances, so this doesn't tell you anything about which clustering is better.
Regarding the silhouette (I'll use ASW for the average silhouette width), the situation is more complicated. These are scaled so that in principle they could be compared between clusterings based on different distances, however note that quality of the clustering (as measured by ASW) is different from the appropriateness of the distance for the application.
It may be that in reality in fact the data are not meaningfully clustered but pretty homogeneous, or rather that their variation doesn't manifest itself in a grouping. In this case a worse clustering in the sense of ASW on the more appropriate distance can be more appropriate (in the sense of giving you better information about the distribution of players) than a strong clustering (ASW) on a less appropriate distance.
The thing to understand here is that the distance should not be chosen in order to optimise the clustering quality.
I give two examples to illustrate that:
(1) If you really want a distance that optimises clustering quality, and, say, you want 5 clusters, the best thing to do would be to assign the distance so that there are five clusters with distance of 1 between each two points in different clusters and zero within each cluster. This would produce a maximum ASW of 1, and chances are you can find a weird nonlinear transformation of the original variables that delivers you such a thing or something close to it, however nobody would think that this is a good way to arrive at a meaningful clustering of the data. Therefore optimisation of the ASW by choice of distance is useless.
(2) Let's say you have 20 variables, and the first one, V1, is measured so that it has a variance of 1000, whereas the 19 others have a variance of 1. Let's also say that if you look at the variables separately, V1 by accident is the one that is most clearly clustered, however there is also (if weaker) clustering structure in the other variables, that at least in part differs from the clustering in V1. If you don't standardise, V1 will dominate the clustering totally and the other 19 variables will have an impact pretty close to zero on the clustering, achieving a good ASW. If you standardise, what you implicitly do is to weight V1 down and the others up, so that the potential impact of the variables on the clustering is unified. In fact, V1, having the strongest clustering structure, will still probably have the most impact, but that may mean 10% or so, rather than almost 100% without scaling. The ASW will probably be worse because the overall clustering structure is not as pronounced as on V1 alone, but surely if you collect 20 variables and all of them are meaningful and important from a subject-matter point of view, you don't want a clustering that is essentially based on one variable alone! You should rather accept the somewhat less sharp grouping that you get taking all your variables into account, which just reflects the overall information more appropriately than a clustering based on one variable only, even though the latter has the better ASW.
(Note that nothing of this discussion stops you from using the ASW to compare clusterings based on the same distance.)
