1
$\begingroup$

I am trying to do cluster analysis for a very small data set (<100) in higher dimensional feature space. I tried K-means and Hierarchical clustering, but I found no 'elbow' and also the silhouette score was never above 15%. To be honest, I do not know if I could use other algorithms as I am not sure which is best for my use case.

I tried to use PCA, but I can come up with examples where the 2D embedding has clusters but not the original feature space, and vice versa.

My question is:

When can I safely conclude that there is no clusters in my data?

Update In this notebook, I try to generate examples where PCA is misleading.

$\endgroup$
9
  • 5
    $\begingroup$ There is no absolute safety here, or almost anywhere else in statistics, but as a rule of thumb I would say that if clear clusters exist they will usually be evident on a plot of PC2 and PC1 scores. Experts will love the challenge to produce counterexamples. $\endgroup$
    – Nick Cox
    Jan 23, 2022 at 15:23
  • $\begingroup$ A plane (meaning a winged vehicle) hiding just beneath another plane might be an image to think about. $\endgroup$
    – Nick Cox
    Jan 23, 2022 at 16:51
  • 1
    $\begingroup$ "cluster analysis" or "clustering", not "clustering analysis", is the right term. $\endgroup$
    – ttnphns
    Jan 24, 2022 at 10:42
  • 3
    $\begingroup$ Theoretically/philosophically, no one can answer your question in a general way, because there is no general definition of what is a cluster: how a "cluster structure" begins to be different from a "no-cluster structure". $\endgroup$
    – ttnphns
    Jan 24, 2022 at 10:47
  • 1
    $\begingroup$ Some internal clustering criteria, such as Gap statistic, attempt to check the "no-cluster" vs "there are clusters" hypothesis. But this is not assumption-free check. $\endgroup$
    – ttnphns
    Jan 24, 2022 at 10:52

1 Answer 1

2
$\begingroup$

Generally, as mentioned in the comments, there are different concepts of what a cluster is, and whether "there are clusters" or not depends on your cluster concept. I wrote something about this as Chapter "Clustering strategy and method selection" in the "Handbook of Cluster Analysis", see here: https://arxiv.org/abs/1503.02059

Here's one possibility to decide that there is no evidence for clustering. This requires you to specify two things, which should be related to the cluster concept of interest:

  1. A null model that models your idea of homogeneous data (for example a Gaussian distribution, or a uniform distribution, as used for the gap statistic). The null model should be chosen so that non-clustering features of your data are matched, for example by using the estimated mean and covariance matrix; see also https://link.springer.com/article/10.1007/s11222-015-9566-5

  2. A statistic that measures the degree of clustering. For $k$-means this can well be the $k$-means objective function, but it might also be something else such as the Average Silhouette Width.

You can then generate data sets from the null model, cluster the data into two (or more) clusters, compute the statistic lots of times (2000, say), which simulates a distribution of the test statistic under the null model, and compare the value(s) that you get on your data to the null distribution. If your data don't give you significantly better values, there is no evidence for clustering.

By the way, this is the idea of the gap statistic (using a uniform null and the log objective function of $k$-means), which gives you a formal rule for choosing "no clustering" (i.e., number of clusters 1) as opposed to the Silhouette Width or the elbow rule.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.