I have a dataset in which two raters (Human raters VS. ML model) assign "0" to pseudowords if they consider it as being masculine or "1" if they think they are feminine. So, it is a binomial dataset. I would like to calculate agreement between these raters in R, but I got weird results when using the function kappa2(data, 'unweighted'). There is something wrong (maybe I've chosen the wrong test) because I got a kappa=0 with p=1 (suggesting total disagreement), but for 56 ratings for each rater, there is only one disagreement between them. In other words, they agree for 55 pseudowords. How come the kappa2() function understand that there is a total disagreement between raters?

my code is below:


Model_answers_o=subset(dt, word_ending=='-o' & easy_diff=='easy', select=c(label_model))
Subject_answers_o=subset(dt, word_ending_pp=='-o' & easy_dif_pp=='easy', 
Table_answers_o=cbind(Model_answers_o, Subject_answers_o)

kappa2(Table_answers_o, 'equal')#Kappa = 0  z = 0  p-value = 1
kappa2(Table_answers_o, 'unweighted') #Kappa = 0  z = 0  p-value = 1

Can anyone let me know which Kappa function and library should I use in R?

Agreement if model and humans assign the same gender to a pseudo word item (both feminine gender), disagreement if model assign masculine but human assign feminine. So, there is nothing to do with model’s accuracy because pseudo words don’t exist so there is no correct/incorrect response.

  • 2
    $\begingroup$ Hard to answer this without seeing the table of frequencies but note that $\kappa = 0$ means there is no agreement over and above what you would expect by chance, it does not mean total disagreement. $\endgroup$
    – mdewey
    Nov 28 '20 at 13:33
  • $\begingroup$ How do you define "agreement"? A simple solution would be "accuracy", i.e. ML and humans agreed on which percentage. You can code it as mean(model_answers == subject_answers) if I understand your variables correctly. $\endgroup$
    – Ott Toomet
    Nov 29 '20 at 1:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.