1
$\begingroup$

I am trying to compute inter-rater reliability for ranking multiple objects across multiple scenarios.

For example suppose for scenario A we had multiple objects and multiple raters like in the table below.

Object   V   W   X   Y   Z
Rater1   2   1   3   4   5
Rater2   1   3   5   2   4
Rater3   4   5   2   1   3

If I only had this one table, I would just compute Kendall's W and be on my way. But what happens if we have multiple scenarios (say B and C in addition to A)? Suppose in each scenario the objects differ and the raters might differ (think Mechanical Turk data). How might I compute reliability in ranking across all scenarios?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.