Clustering leading to visually overlapping clusters on scatterplot

Question

I am dealing with a dataset with 13 features.

After going through some standard scaling and missing data imputation, I use kmeans from sklearn to create clusters.

Now the point is that, although the elbow method indicates a 7 cluster model, when I visualise the results I get a colored cloud that to human eye tells nothing. As you can imagine I do not have much experience with clustering so this results tells me personally that the clustering has failed.

What I want to ask here is that apart from the different metrics that may describe the performance of kmeans, shouldn’t the scatter plot of two features be clear in terms of clusters?

Another approach I have taken is that I performed PCA to extract the 2 most important features however I get that I these two only explain ~35-40% of the variance. Nevertheless when I then run the clustering algorithm only on those two features the clusters are extremely well segregated from each other.

Am I missing something ? Apologies if the context here is not clear but any insight are more than welcome.

Answer 1

k-means will always find k distinct clusters, even if some of the clusters it finds are actually bunched-up together. You always get the prescribed number of clusters, and there's nothing keeping you from getting inconclusive results, even when choosing k using the elbow method.

Having said that, it's also important to keep in mind that you can't visualize your high-dimensional results directly, and therefore judging whether your 7 clusters are "separated" or "bunched up together" may not be as straightforward as you seem to suggest in your question.

Most lower-dimension projections will remove (at least some of) the information about the distance between data points - in the same way that if you see two aircraft just barely miss each other when looking straight up from the ground, it's more likely that in fact there is a significant vertical separation between them, which you cannot perceive from your vantage point.

Consider these alternative/ complementary algorithms to apply on your data in order to better understand the relationships between your clusters:

• The Gaussian Mixture Model is, very roughly speaking, analogous to K-means (in the sense that they both work in the Euclidean space). Its outputs include a vector of cluster affiliation probabilities for each data point. So, if two or more clusters are less distinct and lie close to each other, you will see many data points splitting their probabilities among those clusters.

• The Mean Shift algorithm is another clustering algorithm, similar to K-means in some respects, but it allows cluster centroids to merge together in the training process. Therefore, with this algorithm, if you start with a "too high" number of clusters for your data, your results may show a number of clusters smaller than the initial k.

Both models are available from libraries in both R and Python.