5
$\begingroup$

I understand the clustering algorithms are usually considered as "unsupervised algorithms", which means they can function in the absence of a response variable, making them applicable in situations where supervised algorithms are not.

This being said, I had the following question about the rise in popularity about clustering algorithms.

Based on some readings I have done, it seems that no matter how sophisticated the clustering algorithms are (e.g. Gaussian Mixture Clustering, HDBSCAN, OPTICS, etc.) - none of these algorithms can provide clear indication as to "what do observations in the same cluster have in common"? In all of these cases, the user has to perform exploratory data analysis on the covariates of all observations assigned to the same cluster, and hope that observations in the same cluster share some defining feature that is unique to their cluster relative to the other clusters.

This means that a clustering algorithm can successfully find clusters in which observations within the same cluster have high levels of homogeneity - but there still might not exist a simple "defining factor" that characterizes the cluster - which can have problems for interpretation. I suppose if there were only three covariates, a visualization of the cluster assignments could be made to try and better understand the clustering assignments ; also, dimensionality reduction techniques like PCA, tSNE and UMAP can be used to visualize the clustering assignments for high dimensional data, but the distortion because of the data compression along with the new convoluted axis of the dimensionality reduction still does not make the earlier question easier to answer: what are the defining factors that characterize each cluster? For example, if we were to run a clustering on the Iris Dataset, we could take the average value of the sepal length, sepal width, petal length, petal width of each cluster - then we could say that average sepal length of all flowers in the first cluster is 5.1 cm and the average sepal length of all flowers in the second cluster is 2.1 cm. However, unless the averages for all variables for all clusters had very low variance and were significantly different from one another, it would still be difficult to characterize clusters in terms of defining factors according to their covariates.

If I understand correctly, one of the main applications of clustering was to facilitate statistical modelling by finding low-variance clusters in the data. For instance, a single regression model on the height/weight data of adults and children might perform worse than two separate regression models for adults and children separately. In this case, clustering algorithms would try to infer that the data contains two more-or-less distinct "clusters" - this could be useful if we knew nothing about the data, but it would still be a challenge to characterize these clusters in terms of the covariates. For example, if there were many variables, even a successful clustering algorithm (e.g. good Rand Index, Silhouette Scores, etc.) still might not be able to provide explanations for these clusters. You might be able to fit regression models on both of these clusters that still performs better than a regression model on the entire data set, but when you wanted to make predictions on new data, you wouldn't know which of the two regression models to use!

This leads me to the next point : With all of these issues identified with clustering algorithms, why are they considered so useful and popular? Is it because we only hear about their success stories, i.e. uses when their happened to be a very distinct cluster structure within the data?

The final point that I wanted to bring up is related to "Rule Based Clustering" and "Fuzzy Clustering". Algorithms like "Association Rule Mining" have the ability to operate in unsupervised environments and still return interpretable criteria for the "clusters" they identify, whereas Decision Trees (e.g. CART) have the ability to provide fully interpretable "rules" (i.e. clusters) in supervised environments. I have seen some uses of "Bayesian Networks" which attempt to learn casual structures within the data - these casual structures and the conditional probability tables that they produce can be used to "carve out clusters" if the certain levels within different variables display high joint probabilities. Finally, I have heard about some redesigns for existing clustering algorithms (e.g http://interpretable-ml.org/icml2020workshop/pdf/06.pdf) which specifically target this issue of cluster interpretability:

enter image description here

Can someone please provide any comments on this? Why did clustering algorithms become so popular when understanding the nature of each cluster might not have exact interpretations?

Thanks!

$\endgroup$
9
  • 7
    $\begingroup$ Cause as Fox Mulder said: "I want to believe". Slightly more seriously this is known as the clustering illusion and it is a relatively common cognitive bias. $\endgroup$
    – usεr11852
    Commented Jan 1, 2022 at 11:21
  • $\begingroup$ I like that quote - Thanks! $\endgroup$
    – stats_noob
    Commented Jan 2, 2022 at 2:33
  • 2
    $\begingroup$ Your issue seems to be that clustering might not produce meaningful results if there are not actually distinct clusters that have different feature profiles. But that limitation is true of supervised methods as well, it's not going to work well if the different classes are not very distinct in any of their features. Both approaches assume that there are distinct groups defined by their features, and in both cases we can describe a group by its differential features, whether the groups are defined by clustering or a pre-defined supervised label. $\endgroup$ Commented Apr 28, 2022 at 16:23
  • $\begingroup$ Semi-supervised learning is very useful/interesting and a lot of it has similarities with generative clustering (e.g. Gaussian Mixture Models). $\endgroup$ Commented Apr 28, 2022 at 16:23
  • 4
    $\begingroup$ This is a somewhat strange question. You seem to think that interpretability is the most important aspect of such algorithms, but this is not generally the case (as for example the popularity of neural network shows). The obvious answer to your question is that those algorithms are popular because many people value other aspects more than interpretability. $\endgroup$
    – J. Delaney
    Commented Apr 28, 2022 at 16:37

2 Answers 2

5
+25
$\begingroup$

Some vague thoughts:

"what do observations in the same cluster have in common?"

You can take two views of clustering. One is to find groups of data points with attributes that are similar in some way. The other is as a generative model, in which case the things that observations in the same cluster have in common is the process that generated the data (Gaussian clustering is normally of this nature). So for example, if we were to cluster the penguin dataset we might identify two clusters, and one would contain most of the Gentoo penguins and the other would be a mix of Adelie and Chinstrap penguins. The thing that gives rise to these clustered variations in flipper length and body size is a biological difference in the populations that generated the data. Adding more variables might allow clustering algorithms to distinguish Adelie penguins from Chinstrap penguins

enter image description here

In all of these cases, the user has to perform exploratory data analysis on the covariates of all observations assigned to the same cluster, and hope that observations in the same cluster share some defining feature that is unique to their cluster relative to the other clusters.

That is rather the use of clustering, as a tool for exploring the data, so we can discover the differences in the data generating procedures by e.g. scientific means. Discovering the data generating processes by scientific means alone might be much more difficult.

For example, if we were to run a clustering on the Iris Dataset, we could take the average value of the sepal length, sepal width, petal length, petal width of each cluster - then we could say that average sepal length of all flowers in the first cluster is 5.1 cm and the average sepal length of all flowers in the second cluster is 2.1 cm.

This is slight missing the point for me. The clustering allows you to see if the data have a cluster structure (or can be explained as a group of overlapping sub-populations), that you can then investigate to see if there may be real differences in the generative process (e.g. whether the different groups of penguins/irises actually are different species). They are a tool for learning the underlying structure of the data to help us understand it. The centroid of the group may not be representative of the whole cluster, even if it is all generated by the same process.

" but when you wanted to make predictions on new data, you wouldn't know which of the two regression models to use!"

This is why I prefer clustering algorithms with soft assignment, as you can then use a weighted combination of the two models, depending on the attributes of the test case.

BTW if you are interested in looking for differences in the data generating process, the clusters don't necessarily need to be distinct and well-separated from each other to be useful. There may just be an excess of observations in a small within the normal range of variation, and identifying that sub-population may be useful, and e.g. Gaussian Mixture Models can do that for you.

Personally I only really like the non-generative clustering methods, where often you have soft-assignment of observations to clusters. I don't really see the point in hard-clustering algorithms in statistics (they are useful in engineering, c.f. vector quantisation).

$\endgroup$
5
$\begingroup$

It is sort of a loaded question to ask why cluster analysis is so popular. This frames cluster analysis as being more popular than it should be. But, you do not explain with specific examples a use of cluster analysis that is not right.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.