Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I found a question that is related here, but it doesn't really goes on what I want to know.

I found a couple of papers using Kappa Statistic from 2006, and 2010, but afterwards I found other authors disagreeing with it like here and here. I also noticed other tests being listed and suggested like the ICC on the question pointed on cross validated I pointed, and on the wiki.

To narrow down the context: I have a set n of participants which consistently were asked to say 'yes/no/maybe' on a set of questions. I've seen authors also changing the yes/maybe to 'evidence' and converting it to yes/no by just summing the quantities of yes and maybe.

My questions thus are:

  1. What is the most recent/accepted or referenced paper in respect to this discussion?
  2. Given my dataset constraints, which test would be more appropriated? I was considering Fleiss but when I saw it was a generalization of the pi-statistic and that the Kappa came out of criticism to pi i got confused. If you could suggest an R function that could do it I would also appreciate.

Thank you.

share|improve this question

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.