# Measure of rater agreements for rank order?

If k raters are asked to rate the same set of objects on a continuous or Likert scale, there is the ICC3 for measuring the inter-rater agreement.

Is there also an agreement measure, if all raters have to order the rated objects by preference?

A naive approach would be to compute the Spearman correlation for all pairs of objects and then take the average, but as this most certainly is a standard problem, I wonder whether there is a standerd solution for it.

• Paired preference models, for instance, or log-linear approaches..
– chl
Nov 21 '20 at 15:14
• @chl These are models that yield score values from ranking data. This is, however, not an issue in my case, because all raters rate all objects completely, and thus ML parameter estimation is not necessary. I am looking for an index that measures how well the raters agree. Nov 21 '20 at 15:52
• Something like the coefficient of concordance Kendall's W, then?
– chl
Nov 21 '20 at 18:40
• @chl Yes, thanks! Kendall's W is exactly what I was looking for. Interestingly, according to the wikipedia site it is almost the same as my suggestion of computing the average Spearman correltaion between all pairs. Nov 21 '20 at 20:02
• @chi I have treid Kendall's W, but the result does not look very reasonable in various test cases (see my answer below). Do yo know any other indices which I might try out? Nov 26 '20 at 10:31

> library(irr)
> kendall(x)$value [1] 0.36  Does someone know of a different index that yields a more reasonable result in this case? • The problem comes from the fact that you're using a 3-point rating scale, which implies there's little variation around the average ratings. The same applies in the case of the ICC for agreement. I can increase you Kendall's W by simply using a larger range of responses, e.g., x <- data.frame(R1=c(1,2,5), R1=c(1,2,4), R1=c(1,2,5), R1=c(3,2,3), R1=c(1,2,5)). – chl Nov 26 '20 at 11:15 • Hm, as these are ranks, gaps in the responses are not possible. Maybe Kendall's W is not appropriate for rankings, but only for Likert scales? Nov 26 '20 at 12:12 • The result isn't unreasonable, and Kendall's W is appropriate. The Spearman correlation between c(1, 2, 3) and c(3, 2, 1) is -1. Out of all the choose(5, 2) = 10 Spearman correlations between the pairs of ranks, 4 of them involving your 4th rater are -1, and the remaining 6 pairs of ranks are correlated 1 with each other. We therefore find a mean Spearman correlation of .2. The Kendall's W is linearly related to the average Spearman correlation for all pairs of ranks. Given$m$judges (here, 5), we can see$\bar{\rho}=((m*W) - 1)/(m - 1)=((5 * .36) - 1)/(5 - 1)=.2\$ Nov 26 '20 at 12:58