1
$\begingroup$

I'll try to be clear by describing the data and the problem. This is for a systematic review of a specific research literature. All articles can be either 'included' or 'excluded'. However for the exclusion criterion there are multiple categories - i.e. articles can be excluded on the basis of an unsuitable sample, poor quality evidence, the wrong kind of intervention, etc. So, not only can the raters agree/disagree on including or excluding an item, but the 'exclusions' could be for different reasons. It is easy enough to use Kappa to calculate agreement for the binary 'include/exclude', but A: Is there a way to compare ratings for the multiple categories for 'exclude' and B: Is this even necessary?

P.S. I have read through similar questions on the site and didn't feel that my question was answered, but if the answer is indeed addressed elsewhere I'd be happy to be redirected. Thanks.

$\endgroup$

1 Answer 1

1
$\begingroup$

As far as your question A is concerned it is fine to compute Cohen's $\kappa$ for more than two categories using the usual formula or your favourite statistical software. There may be some problems doing this if one rater did not use one category at all, software differs, but you can do a limited amount of collapsing of categories to avoid it.

As far as question B this is really a matter for you and the science underlying the problem. Does it matter too much why each was excluded? I think the majority view would be that it does not matter.

$\endgroup$
1
  • $\begingroup$ Thank you mdewey. A: Thanks for confirming this. I thought this was possible. Yes, there are categories not used by particular raters; will look into 'collapsing' though. Although saying this as the second cluster are sub-categories of the second binary category and nominal/non-comparable, it may be better to just look at the first binary B: No it probably doesn't matter too much although as the review is configurative or works with a lot of qualitative data, it might have served as an analogue for reliability (Cohen's K comes out at 100% with binary; may seem suspicious or over-simplified) $\endgroup$
    – WG82
    Commented Jul 30, 2018 at 16:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.