2
$\begingroup$

I and another coder want to assess our agreement.

Our experiment involves coding the utterances of participants. there are 17 different codes that can be assigned, and we have independently counted their frequencies.

Participant 1 may have uttered code 1: 3 times, code 2: 2 times, code 3: 1 time, and so on until code 17. participant 2, ditto.

On top of all this, I and the other coder want to compare the frequencies we found.

So for participant 1, coder 1 and coder 2 might agree on the frequency of code 1, and not on code 3, and so on for participant 2 and 3 and 4.

I don't know of a way to ascertain the overall interrater reliability between us- is there a way? and can it be done in SPSS?

Improve this question
$\endgroup$
3
  • $\begingroup$ is there a relationship between the variables? Are they measuring the same construct, or are they independent? $\endgroup$
    – doug.numbers
    Commented May 8, 2013 at 23:10
  • $\begingroup$ hm, it's complicated. they aren't all measuring the same construct. for instance, codes 1-5 all code for a different type of positive verbal information and codes 5-10 are measuring different kinds of negative information. theoretically these meta-categories are independent, but i suppose the individual codes are likely to be interrelated. also, in a correlation matrix i've been finding some correlations between them, but that might be down to a little methodological problem. is that any help? thanks for the reply! $\endgroup$
    – seymour
    Commented May 9, 2013 at 0:19
  • 1
    $\begingroup$ I am not sure what the data means to you. If the variables are independent, then you may need to look at exact agreement for each of the variables. You may find that some of the groups (e.g., variables 1-5 have different agreement than variables 6-10). If some frequencies are rare, then simply percent exact agreement might not be meaningful and you might want to try Cohen's kappa. If the variables can be grouped and if can be summed within groups, then a simple correlation might be meaningful to you. I guess I don't understand the project yet. $\endgroup$
    – doug.numbers
    Commented May 9, 2013 at 3:07

1 Answer 1

Reset to default
2
$\begingroup$

Actually, not in SPSS, but in R, very simply, you could use the package irr. Check this link http://www.cookbook-r.com/Statistical_analysis/Inter-rater_reliability/

Improve this answer
$\endgroup$
1
  • $\begingroup$ It's not clear that the link provides assistance in the case where agreement should be assessed across multiple variables. $\endgroup$
    – John Flournoy
    Commented Aug 19, 2021 at 19:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.