# Inter-rater Reliability for Pairwise Ranking with sparse observations

I would like to calculate inter-rater reliability for data with pairwise comparisons where different pairs are compared for different items by different raters. This doesn't seem to fit the framework of the standard measures (different $\kappa$'s, $\alpha$) because there aren't "categories" that are common across items and the ratings are sparse (the item-rater table is extremely sparse).

In more detail, I have a collection of systems $\{S_i: i\in I\}$ which I want to compare. Each system accepts an input $x$ and produces an output $S_i(x)$, and I want to compare the systems on the average quality of their output. I have a collection of inputs $\{x_j: j\in J\}$, and for each pair of systems $(S_{i_1}, S_{i_2})$, I produce output pairs for a subset of inputs $\{(S_{i_1}(x_j), S_{i_2}(x_j)): j\in J_{i_1, i_2}\}$ where $J_{i_1, i_2}$ is the subset of inputs on which I compare systems $S_{i_1}$ and $S_{i_2}$. Importantly, $\{J_{i_1, i_2}: i_1, i_2 \in I\}$ is a random partition of $J$ so the input sets for different system pairs are disjoint. I then have a group of raters $\{R_k: k\in K\}$ who are assigned "items" i.e. $(S_{i_1}(x_j), S_{i_2}(x_j))$ pairs and must select the better output from the pair. Each rater rates several items and raters are randomly assigned items so that each item receives the same number of ratings.

The table $X_{i_1, i_2, j, k}$ is horrendously sparse for this design. And the categories mean different things for different items. How do I calculate inter-annotator agreement? In particular, what's the appropriate expected agreement since observed agreement is fairly straightforward to calculate.