1
$\begingroup$

I have a clustering problem which I solved using KMeans clustering. I also know that the Elbow Method for cluster evaluation can be used to approximate a feasible pick for the number of clusters.

I see alot of individuals using the cluster inertia for their Elbow Method plot. I also discovered that some like to use the 'Distortion' which should correspond to the average euclidean distance between points in some cluster to its centroid. This can be achieved with something like

distortion = sum(np.min(cdist(X, kmeanModel.cluster_centers_, 'euclidean'), axis=1)) / X.shape[0]

I personally find the distortion metric more intuitive for such an evaluation.

Note that my data is normalized as $(x - \mu)/\sigma$, which aims to make the underlying data roughly normal distributed.

How should I interpret the distortion? My intuition is the following: when I see an average distortion on $k=9$ clusters to be $1.45$, does this now mean that I can expect the average datapoint in the average cluster to be within $1.45$ standard deviations of the cluster centroid?

$\endgroup$
1
$\begingroup$

According to this post, the distortion for $k$ centroids $\vec{\mathrm{m}} = [\mathrm{m}_1, \dots, \mathrm{m}_k]$ and class assignements $\vec{c} = [c_1, \dots, c_n]$ (where $c_i$ is the class found for point $\mathrm{x}_i$) obtained from running k-means on data $\mathcal{D} = \{\mathrm{x}_1, \dots, \mathrm{x}_n\}$ is defined as $$ J(\vec{c},\vec{\mu}) = \sum_{i=1}^n\lVert \mathrm{x}_i - \mathrm{m}_{c_i}\rVert^2$$ So distortion is not the average, however if you multiply it by $1/n$ you do indeed get the average squared distance of a random point to its assigned centroid. Using your values, $\frac{1}{n}J(\vec{c},\vec{\mu}) = 1.45$ tells you that

On average a point is inside a ball of radius $\sqrt{1.45}$ around its centroid

However since your data is normalized, the un-normalized data will we inside an ellipse as the ball is transformed differently along each dimensions by the mapping $\mathrm{x} \mapsto \sigma\odot\mathrm{x} + \mu$.

$\endgroup$
6
  • $\begingroup$ Thanks for your answer. I also interpreted that. However I wonder whether this value corresponds to the standard deviation since the data is normalized? Since you'd say that a z-score of 1,2,3 corresponds to 1,2,3 deviations from the mean. $\endgroup$ – Bjarke Kingo Feb 23 at 13:22
  • $\begingroup$ Is your data multi dimensional or only 1 dimensional ? $\endgroup$ – ArnoV Feb 23 at 13:25
  • $\begingroup$ Its multi-dimensional. I'd have to hold this up to a t-distribution table with correspondings degrees of freedom maybe? $\endgroup$ – Bjarke Kingo Feb 23 at 13:26
  • $\begingroup$ Well then you normalize each dimension separately, so each dimension $i$ has a different mean $\mu_i$ and std $\sigma_i$. Being in a ball of radius $r$ for the normalized data along dimension $i$ means being in an ellipse of radius $\sigma_i r$ for the transformed data. So yes it's an average deviation of $1.45\sigma_i$ in your case, but the deviation depends on which dimension you're considering. $\endgroup$ – ArnoV Feb 23 at 13:31
  • $\begingroup$ I'll change my notation to avoid confusion between the centroid and your grand mean. $\endgroup$ – ArnoV Feb 23 at 13:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.