K means clustering breakup---galaxy spectrum data set

Question

I have a spectrum data set (total 22000). Similar to an electronic wave data, two dimensional (Flux vs Wavelength). A typical set of wavelength plot looks like below

Now I am doing kmeans on this data set to cluster them based on their spectrum shape/pattern only (using sci-kit learn). Some results of the k means clustering is baffling me, I have made a flow chart of how the candidates clustered as I would go on increasing the number of clusters from k=2 to k=5. The flow chart graph of the division of candidates is below (this is on a smaller subset),

Now my most basic question is why is the division of points in clusters happening like it is seen in the plot, as we traverse down the graph. More specifically, why for example in k=4 case in Cluster=1 it makes a group with such a mix bag ? couldnt kmeans instead select it's constituent 1078 or 580 candidates into a single (more cleaner) cluster ? Also is it a coincidence that there is always an identified 13 members group (in golden arrow) (or a 47).

Answer 1

In many real datasets (obviously I don't know about yours), clusters are not well separated, and even if they are, $k$-means clusters will not necessarily correspond to well separated subsets of the data, at least not if the well separated subsets have a covariance structure different from spherical, or varying withing-cluster variation, or if the number of clusters is not well chosen.

$k$-means clusters are defined by within-cluster homogeneity, disregarding separation. This particularly means that just from your flow chart there is no information on whether any of these clusters is well separated. I will call "cluster k.j" cluster no. j in the k clusters solution. Looking at how cluster 4.1 is made up, it may well be that there is no separation between the 580 observations from cluster 3.2 and the 1078 observations from cluster 3.1. Just the fact that these were in different clusters in the three cluster solution doesn't say at all that they need to be in different clusters in solutions with larger $k$.

Imagine a homogeneous dataset, like from a normal distribution, split into $k=2,3,4,\ldots$ clusters in some geometrically balanced fashion; it will always happen that some observations that were together for one $k$ will be split for $k+1$, and partly joined with observations that were in other clusters before. There's not necessarily anything less "clean" about the resulting clustering than what you had at lower $k$.

Regarding your golden arrow 13 observations group, I'm not sure from your explanations of how the diagram was constructed that these are always the same observations. Did you choose the golden arrow in order to show that these are always the same observations? If so, it just means that the 13 observations going from cluster 2.1 to cluster 3.2 remain together also at higher values of $k$. Nothing unusual about that.

There's nothing in the flow chart that looks wrong or surprising to me, and it may just be surprising to you because you may assume implicitly that if k-means gives you a certain solution for a certain $k$, these have to be clear and well separated clusters, which is not the case. And this may either be because your data is so that there are no or only few well separated clusters with the rest of the data homogeneous, or because $k$-means doesn't find separated clusters even if they exist, due to its focus on homogeneity and spherical clusters with roughly equal within-cluster variation. (Not knowing more about data and background I abstain from giving an opinion on whether $k$-means is suitable here or not; it may well not be, as commentators have already indicated.)