1
$\begingroup$

I have obtained a Fliess' Kappa interrater reliability of $37.5\%$ from $90$ randomly selected raters rating $100$ randomly selected, binary items ($Yes/No$).

Question: Is the following interpretation technically correct?

After accounting for the chance-expected agreement, we expect ratings from any 90 raters would genuinely agree with one another on 37.5% of these 100 items.

Note: I'm NOT generalizing to any wider populations of raters or items in my interpretation above. I'm only generalizing to OTHER samples of raters OF THE SAME SIZE.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

In my opinion, it would be unwise to try to generalize from a sample of 2 raters to a larger population of raters. I would either use this as an estimate of how much agreement there is between these two specific raters or, if you really want to generalize, then collect a larger sample of raters. Along similar lines, it would probably be good to collect more than 10 items as well if this is to be published or is important in some other way. Smaller samples are notoriously unrepresentative. Imagine trying to draw conclusions about Americans in general based on data about just two (or even 10) of them. I would also recommend including some quantification of your uncertainty in the estimate, such as a 95% confidence interval. Finally, you may want to read up on generalizability theory, which provides statistical tools for determining how well an estimate may generalize to different facets (e.g., raters or items) and how many raters or items you would need to achieve a desired level of certainty.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks, may I rephrase my question! Suppose 2 raters rated 10 binary items, Fleiss' Kappa is .6 with a 95% CI of [0, 1]. What is the interpretation of this CI? Is the following correct: 95% of the time, after accounting for the chance-expected agreement, we expect ratings from any 2 randomly selected raters of these exact same 10 items would lead to a Kappa between 0 to 1 over long-run repetitions. $\endgroup$
    – rnorouzian
    Jan 1, 2020 at 3:33
  • $\begingroup$ Chapter 5 in Gwet (2014) explains this stuff in great detail. I recommend you read that. Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (4th ed.). Gaithersburg, MD: Advanced Analytics. $\endgroup$
    – Jeffrey Girard
    Jan 1, 2020 at 3:44
  • $\begingroup$ Apologies I can't give a simple answer. As you'll see when you read that chapter, the answer depends on how the CI was calculated (conditional on items, conditional on raters, or unconditional). $\endgroup$
    – Jeffrey Girard
    Jan 1, 2020 at 3:55
  • $\begingroup$ No apologies needed, many thanks, that was what I was looking for! $\endgroup$
    – rnorouzian
    Jan 1, 2020 at 3:57
  • $\begingroup$ Oh and one final note about that chapter. There are some typos in the formulas. So be sure to check the corrections on Gwet's website: agreestat.com/book4/errors_4ed.html $\endgroup$
    – Jeffrey Girard
    Jan 1, 2020 at 4:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.