I am trying to compute inter-rater reliability for ranking multiple objects across multiple scenarios.
For example suppose for scenario A we had multiple objects and multiple raters like in the table below.
Object V W X Y Z Rater1 2 1 3 4 5 Rater2 1 3 5 2 4 Rater3 4 5 2 1 3
If I only had this one table, I would just compute Kendall's W and be on my way. But what happens if we have multiple scenarios (say B and C in addition to A)? Suppose in each scenario the objects differ and the raters might differ (think Mechanical Turk data). How might I compute reliability in ranking across all scenarios?
