2
$\begingroup$

In my system I ask $n$ people to rate [1,5] $m$ objects (e.g. films). I'd like to get a measure of the inter-subject concordance, i.e. how much the users agreed overall, not just on a particular object but on all of the objects.

I know that Kendall's W is a well-established measure of concordance on rankings, but in my case people are not ordering objects, but assigning a 1-5 score. From Kendall's perspective, this results in a lot of ties, making Kendall's W problematic.

What is a suitable measure of inter-subject concordance for ratings rather than for rankings?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

In this case usually the intraclass coefficient is used. It is a measure of interrater agreement, which takes both profiles and elevation into account. There are several ICCs; in your case I would recommend to use the ICC(2,k). This is the appropriate index if all raters rate all films, and raters are selected randomly.

Improve this answer
$\endgroup$
1
  • $\begingroup$ The problem is that the ICC uses parametric tests.. $\endgroup$
    – Cesare Camestre
    Commented Jul 22, 2013 at 9:46
0
$\begingroup$

One way of doing it is to compute standard deviation for each film (as e.g. in https://stats.stackexchange.com/a/23254/6552) and average it over films (perhaps using the numbers of voters as the weights).

Other approach is measure average correlation (Pearson or Spearman) between their rates. Advantage (or disadvantage, depending what you want to do) of this approach is that it excludes simple shifts. E.g. if one person likes the same films as the other, but just casts 1 mark lower, it means that they agree.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks! I understand how to use Pearson between 2 users, but what about the whole set of users? I need a global measure of concordance, not just between 2 users. $\endgroup$
    – Mulone
    Commented Feb 21, 2012 at 22:48
  • $\begingroup$ Use it between all pairs of users (now you got $n(n-1)/2$ correlation coefficients), then average it. $\endgroup$
    – Piotr Migdal
    Commented Feb 21, 2012 at 23:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.