What is the best inter-rater agreement test for Likert scale type questions? As far as I see, Cronbach's $\alpha$ is for internal consistency, or it shows how good items are related to describe the main question. I want to measure inter-rater agreement. Is Cronbach's $\alpha$ the right metric for doing this?
Krippendorff's alpha, originally developed in the field of content analysis, is well-suited for dealing with ordinal ratings such as Likert-scale ratings. It has several advantages over some other measures such as Cohen's Kappa, Fleiss's Kappa, Cronbach's alpha: it is capable of dealing with more than 2 raters; it is robust to missing data; and it can handle different types of scales: nominal, ordinal, etc.
It also accounts for chance agreements better than some other measures like Cohen's Kappa.
Calculation of Krippendorff's alpha is supported by several statistical software packages, including R (by the
irr package), SPSS, etc.
Below are some relevant papers, that discuss Krippendorff's alpha including its properties and its implementation, and compare it with other measures:
Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1), 77-89.
Krippendorff, K. (2004). Reliability in Content Analysis: Some Common Misconceptions and Recommendations. Human Communication Research, 30(3), 411-433. doi: 10.1111/j.1468-2958.2004.tb00738.x
Chapter 3 in Krippendorff, K. (2013). Content Analysis: An Introduction to Its Methodology (3rd ed.): Sage.
There are some additional technical papers in Krippendorff's website.