I and another coder want to assess our agreement.

Our experiment involves coding the utterances of participants. there are 17 different codes that can be assigned, and we have independently counted their frequencies.

Participant 1 may have uttered code 1: 3 times, code 2: 2 times, code 3: 1 time, and so on until code 17. participant 2, ditto.

On top of all this, I and the other coder want to compare the frequencies we found.

So for participant 1, coder 1 and coder 2 might agree on the frequency of code 1, and not on code 3, and so on for participant 2 and 3 and 4.

I don't know of a way to ascertain the overall interrater reliability between us- is there a way? and can it be done in SPSS?

  • $\begingroup$ is there a relationship between the variables? Are they measuring the same construct, or are they independent? $\endgroup$ – doug.numbers May 8 '13 at 23:10
  • $\begingroup$ hm, it's complicated. they aren't all measuring the same construct. for instance, codes 1-5 all code for a different type of positive verbal information and codes 5-10 are measuring different kinds of negative information. theoretically these meta-categories are independent, but i suppose the individual codes are likely to be interrelated. also, in a correlation matrix i've been finding some correlations between them, but that might be down to a little methodological problem. is that any help? thanks for the reply! $\endgroup$ – seymour May 9 '13 at 0:19
  • $\begingroup$ I am not sure what the data means to you. If the variables are independent, then you may need to look at exact agreement for each of the variables. You may find that some of the groups (e.g., variables 1-5 have different agreement than variables 6-10). If some frequencies are rare, then simply percent exact agreement might not be meaningful and you might want to try Cohen's kappa. If the variables can be grouped and if can be summed within groups, then a simple correlation might be meaningful to you. I guess I don't understand the project yet. $\endgroup$ – doug.numbers May 9 '13 at 3:07

Actually, not in SPSS, but in R, very simply, you could use the package irr. Check this link http://www.cookbook-r.com/Statistical_analysis/Inter-rater_reliability/


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.