I have 6 similarity matrices reflecting the proximity of 100 pairs of sounds. Similarity indices are in the range [0,1], with higher values indicating sounds perceived as more alike. What is the most appropriate metric I can use to estimate the inter-rater reliability between them? Thanks a lot.
$\begingroup$
$\endgroup$
1
-
3$\begingroup$ Still asking for a Matlab solution? In this case, I let you add this tag. And feel free to add or delete the edits I made if they don't exactly reflect your design. $\endgroup$– chlOct 20, 2010 at 20:10
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
My first idea would be to try some kind of cluster analysis (e.g. hierarchical clustering) on each similarity matrix, and compare the classification trees across raters. We can derive a similarity index from all dendrograms, as discussed here, A measure to describe the distribution of a dendrogram, or in this review, Comparing Clusterings - An Overview from Wagner and Wagner.
You benefit from working with already existing distance matrices, thus such methods will really reflect the nature of your data, and you can still derive a single numerical value to quantify the closeness of method-specific assessments. The following article may be interesting, if you need to refer to existing work:
Hamer, RM and Cunningham, JW. Cluster Analyzing Profile Data Confounded with Interrater Differences: A Comparison of Profile Association Measures. Applied Psychological Measurement (1981) 5(1): 63-72.
Another approach would be to apply some kind of Principal Component Analysis on each similarity matrix, and keep only the first principal component (the linear combination of all 100 items that account for the maximum of variance). More precisely, as you work with (dis)similarity indices or a particular distance/proximity metric, it is sometimes referred to as Principal Coordinate Analysis or Multidimensional Scaling (MDS), although PCA and MDS would yield similar results when dissimilarities are defined as euclidean distances. There is a working example in Izenman's book (Modern Multivariate Statistical Techniques, chapter 13, "perceptions of color in human vision", pp. 468-470) and a discussion on so-called all-pairs design pp. 471-472. You can then compare the 6 linear combinations (i.e., the weights associated to each sound by rater-specific MDS) to assess their consistency across raters. There, an ICC (as described in my previous answer) could make sense, but I don't know of any application of it in this particular case.
