2
$\begingroup$

For a microarray experiment with ~40,000 probes and ~30 samples I used the clara function from R to cluster my expression matrix. How do I interpret this silhouette plot?

my sil plot!

Firstly, I don't understand how a k of 3 could have the highest sil, considering the algo must be putting together very different genes.

Secondly, many of the clusters for k > 100 have lots of zero and negative scores that is throwing off the average from otherwise tighter clusters (which are the ones I want anyway). How do I improve my choice of k? Is it ok to divide the average silhouette by k? Only take the average of positive silhouettes?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

Silhouette statistic is computed for every object from the set of objects being clustered (what is objects in your case - probes?). Sole objects (objects remained unclustered) in the solution receive silhouette value 0. This of course affects the average silhouette value. You might want to consider quality of clustering only among those objects that were clustered. So, set silhouette value for sole objects to missing value rather than 0 before averaging. This trick implies that sole objects are treated as noise points only and not as clusters on their own. Please be aware I'm not R user and therefore can't comment on clara function.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Yes I'm clustering probes, so this definitely jives with my goals. I'm going to ignore unclustered probes anyway so its reasonable that their score shouldn't contribute. $\endgroup$
    – zzk
    Commented Oct 7, 2012 at 16:18
  • $\begingroup$ Please, don't forget what I've said: the unclustered probes will continue to "contribute", but as noise points only. I.e. they still "spoil" cluster separatedness, but they themselves aren't clusters anymore. $\endgroup$
    – ttnphns
    Commented Oct 7, 2012 at 17:00
1
$\begingroup$

For microarray and gene expression data, the classic clustering algorithms such as k-means and also CLARA/CLARANS are a bad choice. Because they have no understanding of the domain, where some dimensions may not bear any relevant information. (And in fact, the distance function, where you probably use Euclidean distance, is inappropriate too!)

You really should look into algorithms designed for Gene expression clustering, also known as biclustering. Which is closely related to subspace clustering, except that in biclustering you often are only interested in having a shared trend (or even binary high/low expression), while subspace clustering is more numerical on vector spaces.

I do not think using the Silhouette coefficient is of any use here either. I do not know if there is a modified Silhouette coefficient suitable for subspace clusters.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I am aware of biclustering, but finding coherent subspaces with only 30 samples is not very accurate (from what I understand) or what I really want in my downstream analysis. According to this paper, CLARA is stable at n>=30 for microarray data, which is about what I have. biomedcentral.com/1471-2105/6/S2/S10 $\endgroup$
    – zzk
    Commented Oct 7, 2012 at 16:16
  • $\begingroup$ Setting k=100 when having 30 samples definitely is not sound, because at least 70 will be empty. $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Oct 7, 2012 at 17:22
  • $\begingroup$ I'm clustering probes, not samples, so I don't follow you. $\endgroup$
    – zzk
    Commented Oct 8, 2012 at 15:36
  • $\begingroup$ What is a probe, and what is a sample for you? Not everybody uses the terms the same way, unfortunately. $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Oct 8, 2012 at 16:23
  • $\begingroup$ a probe measuring a gene expression level on a microarray, 1 biological sample per microarray results in a matrix with about 45,000 and 30 columns. I want to cluster rows to reduce data complexity. $\endgroup$
    – zzk
    Commented Oct 8, 2012 at 18:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.