I am trying to compute inter-rater reliability for ranking multiple objects across multiple scenarios.

For example suppose for scenario A we had multiple objects and multiple raters like in the table below.

Object   V   W   X   Y   Z
Rater1   2   1   3   4   5
Rater2   1   3   5   2   4
Rater3   4   5   2   1   3

If I only had this one table, I would just compute Kendall's W and be on my way. But what happens if we have multiple scenarios (say B and C in addition to A)? Suppose in each scenario the objects differ and the raters might differ (think Mechanical Turk data). How might I compute reliability in ranking across all scenarios?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.