# Kendall's W for multiple objects and multiple surveys

I am trying to compute inter-rater reliability for ranking multiple objects across multiple scenarios.

For example suppose for scenario A we had multiple objects and multiple raters like in the table below.

Object   V   W   X   Y   Z
Rater1   2   1   3   4   5
Rater2   1   3   5   2   4
Rater3   4   5   2   1   3


If I only had this one table, I would just compute Kendall's W and be on my way. But what happens if we have multiple scenarios (say B and C in addition to A)? Suppose in each scenario the objects differ and the raters might differ (think Mechanical Turk data). How might I compute reliability in ranking across all scenarios?

-