2
$\begingroup$

Suppose 3 datasets has 3 ,7, 4 clusters in their respective dataset.

When I aggregated them as one dataset what's the safest number of cluster to choose as perimeter for kmeans or any supervised clustering?

Already tried by considering average(3,7,4) but not giving me near to good solution as the dataset can vary with dimensions, number of samples etc.

$\endgroup$
0
$\begingroup$

It depends on the degree of overlap between the clusters across datasets.

For example, imagine data points fall within the clusters below, which are colored according to the dataset: red for dataset 1 (3 clusters), green for dataset 2 (7 clusters), blue for dataset 3 (4 clusters). The clusters might be completely disjoint (left plot), giving the maximum of 14 clusters in the aggregated data. At the opposite extreme (middle plot), the clusters might overlap such that they merge into a single cluster in the aggregated data. Or, there might be partial overlap (right plot), yielding some intermediate number of clusters in the aggregated data.

enter image description here

So, assuming the initial clusters are all correctly identified, the aggregated data could contain anywhere between a single cluster and 14 clusters (the total number of initial clusters). The number can't be narrowed down beyond that without further assumptions.

$\endgroup$
3
  • $\begingroup$ What kind of assumptions can help or be considered to reduce the number of clusters rather than considering the max i.e 14 in the above example? $\endgroup$ Sep 20 at 20:14
  • $\begingroup$ @ShihabUllah As an example, suppose the clusters present in each dataset were sampled from a global set of well-separated clusters. In this case, clusters wouldn't 'blur' together (as in the middle plot) when the datasets are aggregated. So, the minimum number of clusters would be 7. This is is different than reducing the maxium number of clusters as you asked about, but it's an example of an assumption that would narrow down the range of possibilities (from 1-14 to 7-14). $\endgroup$
    – user20160
    Sep 21 at 0:45
  • 1
    $\begingroup$ I don't think one can realistically expect to identify the number of clusters for the aggregate dataset based only on the number of clusters in the original datasets. Some additional analysis/modeling would be needed, looking at the actual structure of the clusters and/or data. $\endgroup$
    – user20160
    Sep 21 at 0:45
0
$\begingroup$

@user20160's response is correct. You really have no way of knowing the number of clusters the combination will create, because you don't know how those clusters might overlap. I suppose you could try to estimate it by calculating cluster overlap somehow before merging the datasets, but that might be complicated depending on the dimensionality of your data. Alternatively, you could try to answer the question "How many clusters are in this dataset?" using a different clustering algorithm.

Centroid-based clustering algorithms like k-means are generally not the tool for this task.* If your datasets are large, you might consider a density-based algorithm such as DBSCAN or HDBSCAN. If it is quite small (<100), hierarchical clustering might be all you need.

Both of those have their own drawbacks/nuances†, but I believe they would both be more useful to you in trying to solve this problem.

I suppose if you needed to, you could use those algorithms to inform your choice of a value of $k$ to then use in k-means, but I don't exactly know what the implications of that would be, or how useful the result would be beyond the output of the first algorithm.

*If you are set on k-means, I suppose you could analyze all possible values of $k [1, x]$ where $x = \sum(k_i)$ for all of your datasets $i$, but I think you might benefit from other methods.

†Density-based approaches generally require hyperparameter tuning, hierarchical clustering requires you to choose a linkage function and set a cutoff point for pruning the dendrogram

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.