I'm dealing with the output of a study where raters were semi-randomly paired (based on availability) to rate a total of 50 unique but similar scenarios. Each scenario was rated by a different pair of raters (10 total raters), but not in any organized way (e.g. some raters were paired together more than others, some raters rated many more scenarios overall compared to others). I recognize this is not great study design.
In any case, I need to obtain inter-rater reliability stats for this data. Of course, I can calculate the Cohen's kappa and variance between the two raters for any given scenario, but is there any way to reasonably report a summary statistic for the IRR over the 50 different scenarios? Most trivially, I am thinking about averaging the kappa values over the 50 scenarios, and then averaging the variance for those kappas to get a pooled standard deviation.
This seems overly simple to me though. Would this be at all valid? And if it isn't, would there be another, better way to either find the variance for the average of kappas, or another totally different way to report a summary IRR over these 50 separately rated scenarios?