What are possible reasons of clustering failure I am applying K-means and hierarchical clustering to a dataset of gene expression profiles. Both of them fail, in the sense that by plotting the resulting clusters I cannot really identify people belonging to a certain disease status, for instance. Moreover, by using the elbow plot, it is easy to see that there is no right number of clusters.
My question now is the following: what are common, if any, reasons of clustering failure. Can I, a priori, predict if clustering will be, more or less, effective?
If there is a main reason for clustering failure, is there an easy-to-understand way of visualizing the reason behind it?
 A: The main reason is inappropriate data preprocessing.
People tend to assume they can just dump the data into a black box algorithm and get out clusters. That does not work. Because clustering is unsupervised, it is much more sensitive than many supervised approaches.
Before using any of the clustering algorithms, you first need to understand what they do, in particular of what the data needs to be like.
In particular, the methods are very sensitive to data scaling. Neither not using scaling, nor automatic normalization, is usually appropriate. As a user of clustering, you need to first carefully select, scale, and weight all features. A single bad attribute (such as a record ID) can spoil everything. It gets most problematic if you have nonlinear variables, mixed types, and attributes of different kind. In supervised learning, e.g., a SVM can learn much of the feature selection and linear scaling. But without labels, we can't do this in clustering.
A: Maybe the assumptions of the k-means algorithms are not satisfied. K-means algorithm is making the following assumptions:


*

*Clusters are spatially grouped — or “spherical”

*Clusters are of a similar size

A: Preprocessing and Data analysis is important. You should normalize data, of you don't do it, your KMeans will be influenced a far away more from huge data (ex. Number of car for family and tor. kilometers per year done by a family). In addition, as always, remember to not include categorical variable, it doesn't make any sense, because they are not numbers. Furthermore, if you have many attributes, it is reasonable to reduce them by PCA or other methods.
Finally, there are so e situations in which KMeans cannot find an optimal solution: outliers, different deviations of clusters (the dispersion is not uniform), not spherical shape of clusters and so on...
I recommend you to read this paper:
https://www.researchgate.net/publication/308662786_What_to_Do_When_K-Means_Clustering_Fails_A_Simple_yet_Principled_Alternative_Algorithm
