interrater reliability for multiple variables

I and another coder want to assess our agreement.

Our experiment involves coding the utterances of participants. there are 17 different codes that can be assigned, and we have independently counted their frequencies.

Participant 1 may have uttered code 1: 3 times, code 2: 2 times, code 3: 1 time, and so on until code 17. participant 2, ditto.

On top of all this, I and the other coder want to compare the frequencies we found.

So for participant 1, coder 1 and coder 2 might agree on the frequency of code 1, and not on code 3, and so on for participant 2 and 3 and 4.

I don't know of a way to ascertain the overall interrater reliability between us- is there a way? and can it be done in SPSS?

• is there a relationship between the variables? Are they measuring the same construct, or are they independent? Commented May 8, 2013 at 23:10
• hm, it's complicated. they aren't all measuring the same construct. for instance, codes 1-5 all code for a different type of positive verbal information and codes 5-10 are measuring different kinds of negative information. theoretically these meta-categories are independent, but i suppose the individual codes are likely to be interrelated. also, in a correlation matrix i've been finding some correlations between them, but that might be down to a little methodological problem. is that any help? thanks for the reply! Commented May 9, 2013 at 0:19
• I am not sure what the data means to you. If the variables are independent, then you may need to look at exact agreement for each of the variables. You may find that some of the groups (e.g., variables 1-5 have different agreement than variables 6-10). If some frequencies are rare, then simply percent exact agreement might not be meaningful and you might want to try Cohen's kappa. If the variables can be grouped and if can be summed within groups, then a simple correlation might be meaningful to you. I guess I don't understand the project yet. Commented May 9, 2013 at 3:07