Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

In the case of the kappa-value there are some attempts to qualify how good or bad the agreements are. For example Landis & Koch in the article The Measurement of Observer Agreement for Categorical Data talks about "strength of agreement" based on kappa values:

 Kappa       Strength of agreement
=====       =======================
0.0-0.20     Slight
0.21-0.40    Fair
0.41-0.60    Moderate
0.61-0.80    Substantial
0.81-0.90    Almost perfect

My question is if there are some attempts to qualify the strength of agreement" based on "quadratic weighted kappa values" in the same way. Any references about attempts to define "generic strength of agreement" using quadr. weighted kappa?

My assumption is that it would not be meaningful to define "strenght of agreement" based on kappa and to use the same qualification using quadratic weighted kappa. They are different values.

share|improve this question
add comment

1 Answer

Any such scale is, of course, relatively arbitrary; their purpose is to give readers an intuitive feel for the measure. But I think a better way to do this is to show some intuitively clear display of agreement; the nature of this display would depend on how many raters and perhaps on other factors as well. If there are only two raters a crossabulation works well.

Even unweighted kappa can have a maximum possible value less than 1 if the distribution of ratings is different in different observers. Whether this is good or not depends on your purposes.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.