1
$\begingroup$

I'm dealing with the output of a study where raters were semi-randomly paired (based on availability) to rate a total of 50 unique but similar scenarios. Each scenario was rated by a different pair of raters (10 total raters), but not in any organized way (e.g. some raters were paired together more than others, some raters rated many more scenarios overall compared to others). I recognize this is not great study design.

In any case, I need to obtain inter-rater reliability stats for this data. Of course, I can calculate the Cohen's kappa and variance between the two raters for any given scenario, but is there any way to reasonably report a summary statistic for the IRR over the 50 different scenarios? Most trivially, I am thinking about averaging the kappa values over the 50 scenarios, and then averaging the variance for those kappas to get a pooled standard deviation.

This seems overly simple to me though. Would this be at all valid? And if it isn't, would there be another, better way to either find the variance for the average of kappas, or another totally different way to report a summary IRR over these 50 separately rated scenarios?

$\endgroup$
  • 1
    $\begingroup$ Would Krippendorff's alpha possibly be applicable? Every "unit" of analysis was only rated by two raters each. $\endgroup$ – andrew.stanford Jan 17 '18 at 23:50
0
$\begingroup$

For this kind of study using classical test theory Generalizability Theory is best. A software called EduG is commonly used.

$\endgroup$
  • $\begingroup$ Would generalizability theory work for nominal data? In this study, raters were asked to identify behaviors occurring in a scenario from a list. $\endgroup$ – andrew.stanford Jan 17 '18 at 23:06
  • $\begingroup$ I am mostly ignorant of G-theory because it has been supplanted by Item Response Theory. Perhaps your data will fit an IRT model for multiple raters. I mentioned G-theory because it seems like you are most familiar with a classical test theory framework $\endgroup$ – Jay Schyler Raadt Jan 17 '18 at 23:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.