What is the best inter-rater agreement test for Likert scale type questions? As far as I see, Cronbach's $\alpha$ is for internal consistency, or it shows how good items are related to describe the main question. I want to measure inter-rater agreement. Is Cronbach's $\alpha$ the right metric for doing this?

  • 2
    $\begingroup$ Alpha is not the right tool for this. One method is weighted kappa, but there are others as well see this page for a lot of resources $\endgroup$
    – Peter Flom
    May 23 '14 at 19:07
  • 1
    $\begingroup$ A very nice article which might be useful is Hallgren (2012): tqmp.org/Content/vol08-1/p023/p023.pdf $\endgroup$ May 23 '14 at 19:26
  • $\begingroup$ Paper was very helpful (ICC is the right metric). Thank you $\endgroup$ May 24 '14 at 11:46

Krippendorff's alpha, originally developed in the field of content analysis, is well-suited for dealing with ordinal ratings such as Likert-scale ratings. It has several advantages over some other measures such as Cohen's Kappa, Fleiss's Kappa, Cronbach's alpha: it is capable of dealing with more than 2 raters; it is robust to missing data; and it can handle different types of scales: nominal, ordinal, etc.

It also accounts for chance agreements better than some other measures like Cohen's Kappa.

Calculation of Krippendorff's alpha is supported by several statistical software packages, including R (by the irr package), SPSS, etc.

Below are some relevant papers, that discuss Krippendorff's alpha including its properties and its implementation, and compare it with other measures:

  1. Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1), 77-89.

  2. Krippendorff, K. (2004). Reliability in Content Analysis: Some Common Misconceptions and Recommendations. Human Communication Research, 30(3), 411-433. doi: 10.1111/j.1468-2958.2004.tb00738.x

  3. Chapter 3 in Krippendorff, K. (2013). Content Analysis: An Introduction to Its Methodology (3rd ed.): Sage.

There are some additional technical papers in Krippendorff's website.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.