I am trying to employ the silhouette index in order to decide what number of clusters produced from my dataset is most realistic/probable. I'm aware of how to calculate the SI as well as the general theory behind it, but I am wondering how to apply such an algorithm if my clustering procedure produces K clusters along with a "junk" cluster. The junk cluster, in this case, simply refers to data that would not fit well into any of the k clusters (decided by a distance threshold). If my data only produced k > 1 clusters plus the junk cluster, there would be no issue. However, often K = 1. In such a scenario, Bi (avg dissimilarity to any other clusters) is going to be very large given that I would be comparing a well categorized set of data to a mess of odds and ends in the junk cluster (thus producing a rather large average silhouette). On top of that, when more than one K cluster is produced along with the junk cluster, there is going to be at least some degree of similarity between all K clusters (much smaller Bi). This makes it quite difficult for K > 1 to ever produce an average SI that is above that of K = 1 (or so I am thinking)... Then again, I also need to consider the fact that when K increases, the junk cluster is going to become smaller, which means that it is possible for the internal distance of the junk cluster (average Ai) to improve. I can't decide if these effects will even out, or if I am being completely inappropriate by including the junk cluster at all... Anyone with more experience in the clustering biz have any words of advice?

EDIT: I am using wave_clus, which utilizes something called superparamagnetic clustering to sort the data. The junk cluster contains datapoints that would show some similarity to each other, but not nearly as much consistency as you would see in a true cluster. If enough points in the junk cluster are similar to each other based on the feature extraction method wave_clus uses, those points would be more likely (but not always) to be extracted as an additional 'real' cluster and become separate from the junk cluster. This is exactly what happens when the clustering algorithm produces two 'real' clusters rather than one. In some cases, however, it seems that the clustering algorithm lands on a boundary--where it sometimes produces two real clusters, sometimes one. I would like to evaluate which case is more likely by calculating the silhouette index for each case. If it makes any difference, my actual data consists of waveforms/neural spikes recorded from MEA's. The goal is to determine which spikes came from which cell (since each cell typically produces a characteristic waveform).

  • $\begingroup$ What clustering algorithm did you use here? Are the 'junk' patters at all similar to each other? Eg, do you have patterns on different sides of a coherent cluster such that they are each more similar to the cluster than to each other? What are your data & what is the larger goal for the clustering? $\endgroup$ – gung - Reinstate Monica Jul 30 '16 at 0:18

Treat every object in the "junk cluster" to be a cluster of its own, and you should be able to compute silhouette.

I don't know if the results are in any way helpful, because I believe a solution with less noise will be better.

  • $\begingroup$ Hmm... I don't know. If i do that, then each datapoint in the junk cluster is going to have the best internal validity possible. This would mean that an overall average silhouette of all clusters would improve simply from having more datapoints in the junk cluster--which means that k = 1 cluster again would be favored. Thanks for the thoughts though. $\endgroup$ – qualiaMachine Aug 1 '16 at 0:22
  • $\begingroup$ Nevermind. I forgot to take into account the following: if k = 1 when instead it should be 2, then the junk cluster is going to produce many individual clusters with a small distance from each other, which would produce a smaller SI than when K = 2. $\endgroup$ – qualiaMachine Aug 1 '16 at 3:57
  • $\begingroup$ I believe the silhouette of 1-point clusters is defined to be 0? $\endgroup$ – Has QUIT--Anony-Mousse Aug 1 '16 at 5:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.