I have a question regarding clustering. I have a symmetric matrix of 50 specialties (50 X 50) where each cell represents the number of observations related to each combination of specialties. Some combinations in this matrix do not exist in the data (coded with zero frequency) and the diagonal is zero. I would like to perform hierarchical agglomerative clustering to find the structure of these specialties. But before, I need to transform the frequency in each cell into "proximities" (similarities /dissimilarities). Does anyone know of an index of similarities / dissimilarities on the basis of joint frequencies? Another question, what is the accepted clustering method (single linkage, average linkage, ward's method etc.) to analyze these proximities. All suggestions are welcome.
1 Answer
$\begingroup$
$\endgroup$
You can treat your matrix as an adjacency matrix for a graph with 50 nodes (specialties). The common way to compute "proximities" in a graph is Jaccard similarity measure:
$$ JS(A,B) = \frac{\left | neighbors(A)\cap neighbors(B)\right |}{\left | neighbors(A)\cup neighbors(B)\right |} $$
You can also take the weighted version of the same quantity: the sum of observations in specialties which are common neighbors of A and B, divided by the sum of observations in all specialties which are neighbors of either A or B.
In the end you get a 50x50 similarity matrix and can use it in hierarchical clustering or in whatever you want.
