Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 1 down vote favorite

I am trying to compute inter-rater reliability for ranking multiple objects across multiple scenarios.

For example suppose for scenario A we had multiple objects and multiple raters like in the table below. 

Object   V   W   X   Y   Z
Rater1   2   1   3   4   5
Rater2   1   3   5   2   4
Rater3   4   5   2   1   3

If I only had this one table, I would just compute Kendall's W and be on my way. But what happens if we have multiple scenarios (say B and C in addition to A)? Suppose in each scenario the objects differ and the raters might differ (think Mechanical Turk data). How might I compute reliability in ranking across all scenarios?

share|improve this question

Know someone who can answer? Share a link to this question via email, Google+, Twitter, or Facebook.

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.