1
$\begingroup$

Apologies if this is answered elsewhere. I looked through other posts but I don't think my question is answered.

I have a situation where 2 coders rate whether a series of research papers contain particular elements. The issue is that they can choose any number of elements (we offer n possible elements to choose from, but they simply code if they think each category is present and it is up to them how many categories are present in total).

So coder 1 might indicate that 1, 4, 6 and 10 are present. While coder 2 might indicate that 1, 3, 6, 9, 10 and 15 are present.

I am not sure how to appropriately analyse the rates of agreement.

OPTION A (which we have done, I think incorrectly) is to have a row for each possible element in each paper (so paper 1 element 1, paper 1 element 2,… paper 1 element n, AND THEN paper 2 element 1, paper 2 element 2 etc). Then we have columns for the raters (multiple raters but only 2 raters work on a particular subset of papers) see image

We were then using this syntax in SPSS to try to calculate kappa (question is not about SPSS but in case it provides useful context)
CROSSTABS
/TABLES=Coder_12 BY Coder_13
/FORMAT=AVALUE TABLES
/STATISTICS=KAPPA
/CELLS=COUNT ROW COLUMN
/COUNT ROUND CELL.

However, for complete agreement this would require the coders to have the same number of elements chosen as well as the same specific elements chosen. I am not sure this is the best approach? It gives us one kappa for the pair of coders for all elements in all papers they both coded.

OPTION B would be to just to compare (for each paper) whether the 2 coders agreed element 1 was present or absent, and then whether element 2 was present or absent etc. This would result in a large number of kappa’s for each pair of coders (the total number of papers multiplied by the total number of possible elements). I am somewhat unsure about this option given the number of kappa’s it generates.

OPTION C - something else that I am not aware of 😃

any advice would be greatly appreciated.

$\endgroup$