How to find the number of clusters when more than one datasets are aggregated as one?

Question

Suppose 3 datasets has 3 ,7, 4 clusters in their respective dataset.

When I aggregated them as one dataset what's the safest number of cluster to choose as perimeter for kmeans or any supervised clustering?

Already tried by considering average(3,7,4) but not giving me near to good solution as the dataset can vary with dimensions, number of samples etc.

Answer 1

It depends on the degree of overlap between the clusters across datasets.

For example, imagine data points fall within the clusters below, which are colored according to the dataset: red for dataset 1 (3 clusters), green for dataset 2 (7 clusters), blue for dataset 3 (4 clusters). The clusters might be completely disjoint (left plot), giving the maximum of 14 clusters in the aggregated data. At the opposite extreme (middle plot), the clusters might overlap such that they merge into a single cluster in the aggregated data. Or, there might be partial overlap (right plot), yielding some intermediate number of clusters in the aggregated data.

So, assuming the initial clusters are all correctly identified, the aggregated data could contain anywhere between a single cluster and 14 clusters (the total number of initial clusters). The number can't be narrowed down beyond that without further assumptions.

Answer 2

@user20160's response is correct. You really have no way of knowing the number of clusters the combination will create, because you don't know how those clusters might overlap. I suppose you could try to estimate it by calculating cluster overlap somehow before merging the datasets, but that might be complicated depending on the dimensionality of your data. Alternatively, you could try to answer the question "How many clusters are in this dataset?" using a different clustering algorithm.

Centroid-based clustering algorithms like k-means are generally not the tool for this task.* If your datasets are large, you might consider a density-based algorithm such as DBSCAN or HDBSCAN. If it is quite small (<100), hierarchical clustering might be all you need.

Both of those have their own drawbacks/nuances†, but I believe they would both be more useful to you in trying to solve this problem.

I suppose if you needed to, you could use those algorithms to inform your choice of a value of $k$ to then use in k-means, but I don't exactly know what the implications of that would be, or how useful the result would be beyond the output of the first algorithm.

_{*If you are set on k-means, I suppose you could analyze all possible values of $k [1, x]$ where $x = \sum(k_i)$ for all of your datasets $i$, but I think you might benefit from other methods.}

_{†Density-based approaches generally require hyperparameter tuning, hierarchical clustering requires you to choose a linkage function and set a cutoff point for pruning the dendrogram}