Will the silhouette formula change depending on the distance metric? I am using Silhouette width to compute the best value for k in k-means. As I am performing document clustering, I am calculating the values of a and b as follows in Python:
a = distance(data[index], centroids[clusters[index]], metric=metric, p=p)
b = min([distance(data[index], c) for i,c in enumerate(centroids) if i != currentindex])
score = float(b - a) / max(a, b) if max(a, b) > 0 else 0.0

The original formula from the Wikipedia page is the following:

I am using a cosine similarity measure to compute the distances and was wondering if this formula needs to be changed or it can be left as is. In the above snippet of code, the function distance computes the cosine similarity metric.
 A: Silhouette statistic works on distances, not similarities. One should revert similarities into distances. The general pass to do it: 1) set diagonal as 0, 2) revert sign of elements, 3) find the smallest element and substract it from each element, 4) set diagonal as 0.
For cosine or correlation there is also a geometrically more correct way: distance = sqrt[2(1-similarity)]; it comes from trigonometric "cosine theorem".
BTW, if you use SPSS you can find a collection of macros on my web-page that compute a number of clustering criterions, including Silhouette.
A: You can calculate the silhouette for similarity matrix. The seminal paper, by P.J Rousseeuw about silhouette, explains about how to calculate silhouette from similarity matrix:


*

*calculation for Cohesion remains same. 

*For computing Separation, take maximum instead of minimum. 

*for calculating silhouete, the numerator changes as follows: cohesion-separation. 


Refer to page 57 in the paper Silhouettes:a graphical aid to the interpretation and validation of cluster analysis, by Peter Rousseeuw
