I understand the clustering algorithms are usually considered as "unsupervised algorithms", which means they can function in the absence of a response variable, making them applicable in situations where supervised algorithms are not.
This being said, I had the following question about the rise in popularity about clustering algorithms.
Based on some readings I have done, it seems that no matter how sophisticated the clustering algorithms are (e.g. Gaussian Mixture Clustering, HDBSCAN, OPTICS, etc.) - none of these algorithms can provide clear indication as to "what do observations in the same cluster have in common"? In all of these cases, the user has to perform exploratory data analysis on the covariates of all observations assigned to the same cluster, and hope that observations in the same cluster share some defining feature that is unique to their cluster relative to the other clusters.
This means that a clustering algorithm can successfully find clusters in which observations within the same cluster have high levels of homogeneity - but there still might not exist a simple "defining factor" that characterizes the cluster - which can have problems for interpretation. I suppose if there were only three covariates, a visualization of the cluster assignments could be made to try and better understand the clustering assignments ; also, dimensionality reduction techniques like PCA, tSNE and UMAP can be used to visualize the clustering assignments for high dimensional data, but the distortion because of the data compression along with the new convoluted axis of the dimensionality reduction still does not make the earlier question easier to answer: what are the defining factors that characterize each cluster? For example, if we were to run a clustering on the Iris Dataset, we could take the average value of the sepal length, sepal width, petal length, petal width of each cluster - then we could say that average sepal length of all flowers in the first cluster is 5.1 cm and the average sepal length of all flowers in the second cluster is 2.1 cm. However, unless the averages for all variables for all clusters had very low variance and were significantly different from one another, it would still be difficult to characterize clusters in terms of defining factors according to their covariates.
If I understand correctly, one of the main applications of clustering was to facilitate statistical modelling by finding low-variance clusters in the data. For instance, a single regression model on the height/weight data of adults and children might perform worse than two separate regression models for adults and children separately. In this case, clustering algorithms would try to infer that the data contains two more-or-less distinct "clusters" - this could be useful if we knew nothing about the data, but it would still be a challenge to characterize these clusters in terms of the covariates. For example, if there were many variables, even a successful clustering algorithm (e.g. good Rand Index, Silhouette Scores, etc.) still might not be able to provide explanations for these clusters. You might be able to fit regression models on both of these clusters that still performs better than a regression model on the entire data set, but when you wanted to make predictions on new data, you wouldn't know which of the two regression models to use!
This leads me to the next point : With all of these issues identified with clustering algorithms, why are they considered so useful and popular? Is it because we only hear about their success stories, i.e. uses when their happened to be a very distinct cluster structure within the data?
The final point that I wanted to bring up is related to "Rule Based Clustering" and "Fuzzy Clustering". Algorithms like "Association Rule Mining" have the ability to operate in unsupervised environments and still return interpretable criteria for the "clusters" they identify, whereas Decision Trees (e.g. CART) have the ability to provide fully interpretable "rules" (i.e. clusters) in supervised environments. I have seen some uses of "Bayesian Networks" which attempt to learn casual structures within the data - these casual structures and the conditional probability tables that they produce can be used to "carve out clusters" if the certain levels within different variables display high joint probabilities. Finally, I have heard about some redesigns for existing clustering algorithms (e.g http://interpretable-ml.org/icml2020workshop/pdf/06.pdf) which specifically target this issue of cluster interpretability:
Can someone please provide any comments on this? Why did clustering algorithms become so popular when understanding the nature of each cluster might not have exact interpretations?
Thanks!