I am trying to employ the silhouette index in order to decide what number of clusters produced from my dataset is most realistic/probable. I'm aware of how to calculate the SI as well as the general theory behind it, but I am wondering how to apply such an algorithm if my clustering procedure produces K clusters along with a "junk" cluster. The junk cluster, in this case, simply refers to data that would not fit well into any of the k clusters (decided by a distance threshold). If my data only produced k > 1 clusters plus the junk cluster, there would be no issue. However, often K = 1. In such a scenario, Bi (avg dissimilarity to any other clusters) is going to be very large given that I would be comparing a well categorized set of data to a mess of odds and ends in the junk cluster (thus producing a rather large average silhouette). On top of that, when more than one K cluster is produced along with the junk cluster, there is going to be at least some degree of similarity between all K clusters (much smaller Bi). This makes it quite difficult for K > 1 to ever produce an average SI that is above that of K = 1 (or so I am thinking)... Then again, I also need to consider the fact that when K increases, the junk cluster is going to become smaller, which means that it is possible for the internal distance of the junk cluster (average Ai) to improve. I can't decide if these effects will even out, or if I am being completely inappropriate by including the junk cluster at all... Anyone with more experience in the clustering biz have any words of advice?
EDIT: I am using wave_clus, which utilizes something called superparamagnetic clustering to sort the data. The junk cluster contains datapoints that would show some similarity to each other, but not nearly as much consistency as you would see in a true cluster. If enough points in the junk cluster are similar to each other based on the feature extraction method wave_clus uses, those points would be more likely (but not always) to be extracted as an additional 'real' cluster and become separate from the junk cluster. This is exactly what happens when the clustering algorithm produces two 'real' clusters rather than one. In some cases, however, it seems that the clustering algorithm lands on a boundary--where it sometimes produces two real clusters, sometimes one. I would like to evaluate which case is more likely by calculating the silhouette index for each case. If it makes any difference, my actual data consists of waveforms/neural spikes recorded from MEA's. The goal is to determine which spikes came from which cell (since each cell typically produces a characteristic waveform).