# Inter-rater agreement for Likert scale

What is the best inter-rater agreement test for Likert scale type questions? As far as I see, Cronbach's $\alpha$ is for internal consistency, or it shows how good items are related to describe the main question. I want to measure inter-rater agreement. Is Cronbach's $\alpha$ the right metric for doing this?

• Alpha is not the right tool for this. One method is weighted kappa, but there are others as well see this page for a lot of resources May 23, 2014 at 19:07
• A very nice article which might be useful is Hallgren (2012): tqmp.org/Content/vol08-1/p023/p023.pdf May 23, 2014 at 19:26
• Paper was very helpful (ICC is the right metric). Thank you May 24, 2014 at 11:46

Krippendorff's alpha, originally developed in the field of content analysis, is well-suited for dealing with ordinal ratings such as Likert-scale ratings. It has several advantages over some other measures such as Cohen's Kappa, Fleiss's Kappa, Cronbach's alpha: it is capable of dealing with more than 2 raters; it is robust to missing data; and it can handle different types of scales: nominal, ordinal, etc.

It also accounts for chance agreements better than some other measures like Cohen's Kappa.

Calculation of Krippendorff's alpha is supported by several statistical software packages, including R (by the irr package), SPSS, etc.

Below are some relevant papers, that discuss Krippendorff's alpha including its properties and its implementation, and compare it with other measures:

1. Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1), 77-89.

2. Krippendorff, K. (2004). Reliability in Content Analysis: Some Common Misconceptions and Recommendations. Human Communication Research, 30(3), 411-433. doi: 10.1111/j.1468-2958.2004.tb00738.x

3. Chapter 3 in Krippendorff, K. (2013). Content Analysis: An Introduction to Its Methodology (3rd ed.): Sage.

There are some additional technical papers in Krippendorff's website.