I have a clustering problem which I solved using KMeans clustering. I also know that the Elbow Method for cluster evaluation can be used to approximate a feasible pick for the number of clusters.
I see alot of individuals using the cluster inertia for their Elbow Method plot. I also discovered that some like to use the 'Distortion' which should correspond to the average euclidean distance between points in some cluster to its centroid. This can be achieved with something like
distortion = sum(np.min(cdist(X, kmeanModel.cluster_centers_, 'euclidean'), axis=1)) / X.shape
I personally find the distortion metric more intuitive for such an evaluation.
Note that my data is normalized as $(x - \mu)/\sigma$, which aims to make the underlying data roughly normal distributed.
How should I interpret the distortion? My intuition is the following: when I see an average distortion on $k=9$ clusters to be $1.45$, does this now mean that I can expect the average datapoint in the average cluster to be within $1.45$ standard deviations of the cluster centroid?