2
$\begingroup$

I am applying K-means and hierarchical clustering to a dataset of gene expression profiles. Both of them fail, in the sense that by plotting the resulting clusters I cannot really identify people belonging to a certain disease status, for instance. Moreover, by using the elbow plot, it is easy to see that there is no right number of clusters.

My question now is the following: what are common, if any, reasons of clustering failure. Can I, a priori, predict if clustering will be, more or less, effective?

If there is a main reason for clustering failure, is there an easy-to-understand way of visualizing the reason behind it?

Improve this question
$\endgroup$
3
  • $\begingroup$ Are you sure that the people belonging to certain disease statuses are close (in proximity) in your feature space? Are you using a sensible distance metric for your data? $\endgroup$
    – Dan
    Commented May 30, 2018 at 13:38
  • $\begingroup$ @Dan Well, not really. I tried many things: disease status, region of the brain from where the sample was taken, etc... None of them works. And yes, I tried many different distance metrics. I expected to see some clusters, like normal and 'definite disease', but it failed. $\endgroup$
    – wrong_path
    Commented May 30, 2018 at 13:41
  • $\begingroup$ very related: stats.stackexchange.com/questions/133656/… $\endgroup$
    – Ferdi
    Commented Jun 4, 2018 at 9:13

3 Answers 3

Reset to default
3
$\begingroup$

The main reason is inappropriate data preprocessing.

People tend to assume they can just dump the data into a black box algorithm and get out clusters. That does not work. Because clustering is unsupervised, it is much more sensitive than many supervised approaches.

Before using any of the clustering algorithms, you first need to understand what they do, in particular of what the data needs to be like.

In particular, the methods are very sensitive to data scaling. Neither not using scaling, nor automatic normalization, is usually appropriate. As a user of clustering, you need to first carefully select, scale, and weight all features. A single bad attribute (such as a record ID) can spoil everything. It gets most problematic if you have nonlinear variables, mixed types, and attributes of different kind. In supervised learning, e.g., a SVM can learn much of the feature selection and linear scaling. But without labels, we can't do this in clustering.

Improve this answer
$\endgroup$
0
$\begingroup$

Maybe the assumptions of the k-means algorithms are not satisfied. K-means algorithm is making the following assumptions:

  • Clusters are spatially grouped — or “spherical”
  • Clusters are of a similar size
Improve this answer
$\endgroup$
0
$\begingroup$

Preprocessing and Data analysis is important. You should normalize data, of you don't do it, your KMeans will be influenced a far away more from huge data (ex. Number of car for family and tor. kilometers per year done by a family). In addition, as always, remember to not include categorical variable, it doesn't make any sense, because they are not numbers. Furthermore, if you have many attributes, it is reasonable to reduce them by PCA or other methods.

Finally, there are so e situations in which KMeans cannot find an optimal solution: outliers, different deviations of clusters (the dispersion is not uniform), not spherical shape of clusters and so on...

I recommend you to read this paper:

https://www.researchgate.net/publication/308662786_What_to_Do_When_K-Means_Clustering_Fails_A_Simple_yet_Principled_Alternative_Algorithm

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.