1
$\begingroup$

Not sure if this is best placed here but I will have a go.

I am working with clinical data in order to stratify patients using different biomarkers. I have log transformed and MinMax normalised all variables for the 1200 patients.

After plotting a heatmap, it would seem some individuals do cluster together, and biologically the phenotypes identified are meaningful.

enter image description here

Despite this PCA does not show any such clustering and instead shows a large agglomeration of individuals. K-means identifies the best number of clusters (if any) to be 3 (based on elbow method)

A silhouette plot also indicates that the the patients are very close to/on the neighbouring decision boundaries with an average coefficient of 0.21 with 3 clusters.

enter image description here

My question is - is it ok to separate patients on the basis of these statistics? We would like to separat them as biologically/clinically they behave different, and it seems there may be some difference here too. Equally, it is of course important the the plot is correct and I do not "force" the patients into different groups out of wont.

Any input is much appreciated!

| cite | improve this question | |
$\endgroup$
  • $\begingroup$ Why have you applied min/max rescaling instead of zscores? $\endgroup$ – g3o2 Jul 28 '17 at 9:18
  • $\begingroup$ Just out of choice. I also experimented with tanhEstimation. Would you consider mimax to be wrong? $\endgroup$ – JB1 Jul 28 '17 at 9:22
  • 1
    $\begingroup$ I don't mean to sound like a dork but "unsupervised clustering" is redundant term. You should either call it "unsupervised classification" or simply "clustering". $\endgroup$ – Digio Jul 28 '17 at 9:38
  • $\begingroup$ youre right - changed $\endgroup$ – JB1 Jul 28 '17 at 9:46
  • 1
    $\begingroup$ The terms being supposedly mainstream does not remove the paradox. $\endgroup$ – g3o2 Jul 28 '17 at 18:10
1
$\begingroup$

Don't rely on cluster analysis.

Because a single additional data point often can cause a very different result, unfortunately. Try removing a few samples, then rerun your clustering and compare the results! Judging from your Silhouette, I don't think clustering "worked" on this data (and beware that k-means will always find k clusters, even on uniform data!) - your preprocessing may not yet be good enough.

Treat it as a hypothesis generator instead.

You ran clustering, and you got three clusters. This is now your working hypothesis, that you need to verify with experiments. You likely will also need to formalize it first, before you can do experiments: what is different between these groups, and how can we classify new data into the correct group. I wouldn't be surprised if you find you only need to treat one cluster special, and several clusters should be one.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks @Anony-Mousse. I think this was actually the answer i was looking for to reassure me that i was not missing some vital information. What I have found (possible clusters) appears to be interesting clinically, and it is this that I should explore and use to develop further :) $\endgroup$ – JB1 Aug 22 '17 at 11:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.