43
$\begingroup$

I found this tutorial, which suggests that you should run the scale function on features before clustering (I believe that it converts data to z-scores).

I'm wondering whether that is necessary. I'm asking mostly because there's a nice elbow point when I don't scale the data, but it disappears when it's scaled. :)

$\endgroup$
59
$\begingroup$

The issue is what represents a good measure of distance between cases.

If you have two features, one where the differences between cases is large and the other small, are you prepared to have the former as almost the only driver of distance?

So for example if you clustered people on their weights in kilograms and heights in metres, is a 1kg difference as significant as a 1m difference in height? Does it matter that you would get different clusterings on weights in kilograms and heights in centimetres? If your answers are "no" and "yes" respectively then you should probably scale.

On the other hand, if you were clustering Canadian cities based on distances east/west and distances north/south then, although there will typically be much bigger differences east/west, you may be happy just to use unscaled distances in either kilometres or miles (though you might want to adjust degrees of longitude and latitude for the curvature of the earth).

$\endgroup$
31
$\begingroup$

Other answers are correct, but it might might help to get an intuitive grasp of the problem by seeing an example. Below, I generate a dataset that has two clear clusters, but the non-clustered dimension is much larger than the clustered dimension (note the different scales on the axes). Clustering on the non-normalised data fails. Clustering on the normalised data works very well.

The same would apply with data clustered in both dimensions, but normalisation would help less. In that case, it might help to do a PCA, then normalise, but that would only help if the clusters are linearly separable, and don't overlap in the PCA dimensions. (This example only works so clearly because of the low cluster count)

synthetic clustered data, with k-means clustering on both the normalised and non-normalised versions

import numpy as np
import seaborn
import matplotlib.pyplot as plt
from sklearn.cluster import KMeans

rnorm = np.random.randn

x = rnorm(1000) * 10  
y = np.concatenate([rnorm(500), rnorm(500) + 5])

fig, axes = plt.subplots(3, 1)

axes[0].scatter(x, y)
axes[0].set_title('Data (note different axes scales)')

km = KMeans(2)

clusters = km.fit_predict(np.array([x, y]).T)

axes[1].scatter(x, y, c=clusters, cmap='bwr')
axes[1].set_title('non-normalised K-means')

clusters = km.fit_predict(np.array([x / 10, y]).T)

axes[2].scatter(x, y, c=clusters, cmap='bwr')
axes[2].set_title('Normalised K-means')
$\endgroup$
17
$\begingroup$

It depends on your data.

If you have attributes with a well-defined meaning. Say, latitude and longitude, then you should not scale your data, because this will cause distortion. (K-means might be a bad choice, too - you need something that can handle lat/lon naturally)

If you have mixed numerical data, where each attribute is something entirely different (say, shoe size and weight), has different units attached (lb, tons, m, kg ...) then these values aren't really comparable anyway; z-standardizing them is a best-practise to give equal weight to them.

If you have binary values, discrete attributes or categorial attributes, stay away from k-means. K-means needs to compute means, and the mean value is not meaningful on this kind of data.

$\endgroup$
5
$\begingroup$

As explained in this paper, the k-means minimizes the error function using the Newton algorithm, i.e. a gradient-based optimization algorithm. Normalizing the data improves convergence of such algorithms. See here for some details on it.

The idea is that if different components of data (features) have different scales, then derivatives tend to align along directions with higher variance, which leads to poorer/slower convergence.

$\endgroup$
3
$\begingroup$

Standardization is an important step of Data preprocessing.

it controls the variability of the dataset, it convert data into specific range using a linear transformation which generate good quality clusters and improve the accuracy of clustering algorithms, check out the link below to view its effects on k-means analysis.

https://pdfs.semanticscholar.org/1d35/2dd5f030589ecfe8910ab1cc0dd320bf600d.pdf

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.