1
$\begingroup$

I am looking to calculate Cohen's kappa for multiple items in a scale (15 items, with two raters). The problem is that for 12 of the items the responses available to the raters were binary (yes, no). The remaining three are categorical responses with more than two choices (e.g. cat1,cat2,cat3,cat4). Does it make sense to use the Kappa statistic across both types of variables at the same time? Or should I calculate Kappa separately for the binary variables and then again for the non-binary variables?

Thanks.

$\endgroup$
1
$\begingroup$

The standard form of kappa is for agreement between two categorical variables with the same number of categories (and indeed the same categories so you know which category on variable A matched with which on variable B.

There is a method developed by Brennan and Light in a paper entitled "Measuring agreement when two observers classify people into categories not defined in advance" and available here which deals with the case where the categories used are different and may not have the same number.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.