Quadratic weighted kappa strength of agreement

Question

In the case of the kappa-value there are some attempts to qualify how good or bad the agreements are. For example Landis & Koch in the article The Measurement of Observer Agreement for Categorical Data talks about "strength of agreement" based on kappa values:

 Kappa       Strength of agreement
=====       =======================
0.0-0.20     Slight
0.21-0.40    Fair
0.41-0.60    Moderate
0.61-0.80    Substantial
0.81-0.90    Almost perfect

My question is if there are some attempts to qualify the strength of agreement" based on "quadratic weighted kappa values" in the same way. Any references about attempts to define "generic strength of agreement" using quadr. weighted kappa?

My assumption is that it would not be meaningful to define "strenght of agreement" based on kappa and to use the same qualification using quadratic weighted kappa. They are different values.

Answer 1

Any such scale is, of course, relatively arbitrary; their purpose is to give readers an intuitive feel for the measure. But I think a better way to do this is to show some intuitively clear display of agreement; the nature of this display would depend on how many raters and perhaps on other factors as well. If there are only two raters a crossabulation works well.

Even unweighted kappa can have a maximum possible value less than 1 if the distribution of ratings is different in different observers. Whether this is good or not depends on your purposes.

Answer 2

I have looked for various authoritative opinions for this, but have not found a definitive answer. Rather I did notice that for inter-class correlations, which differ from Cohen's kappa which qualifies IRR based on all-or-nothing-agreement, I see that ICCs incorporate the size of the disagreement to compute IRR such that larger disagreements result in lower ICCS than smaller disagreements. And in this case, for ICC, I found that in Cicchetti (1994) provided "commonly cited cutoffs for qualitative rankings for ICC values, such that IRR being

• poor for ICC values less than .40,
• fair for .40 to .59
• good for .60 to .74
• excellent for values between .75 and 1.0.

[Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment. 1994;6(4):284–290. [Ref list]

So by analogy I think QWK's would be characterized differently than linear Kappa's, and the exact way to quantify these differences might vary depending on the use case. In my experience, characterizing the overall distribution of difference, say with a visual "confusion matrix", can be a helpful way to show IRR. The cited reference and the URL where most of this text was originally from also provide guidance about characterizing IRR.

This text was lifted almost verbatim from Kevin A. Hallgren "Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial" here.