Using Cohen's kappa for multiple item types (e.g. binary and non-binary)

Question

I am looking to calculate Cohen's kappa for multiple items in a scale (15 items, with two raters). The problem is that for 12 of the items the responses available to the raters were binary (yes, no). The remaining three are categorical responses with more than two choices (e.g. cat1,cat2,cat3,cat4). Does it make sense to use the Kappa statistic across both types of variables at the same time? Or should I calculate Kappa separately for the binary variables and then again for the non-binary variables?

Thanks.

Answer 1

The standard form of kappa is for agreement between two categorical variables with the same number of categories (and indeed the same categories so you know which category on variable A matched with which on variable B.

There is a method developed by Brennan and Light in a paper entitled "Measuring agreement when two observers classify people into categories not defined in advance" and available here which deals with the case where the categories used are different and may not have the same number.