Quadratic weighted kappa

Question

I have done a little Googling about quadratic weighted kappa, but I couldn't find a good explanation that make me understood that. Can somebody give some resource or brief explanation?

Answer 1

The Kappa coefficient is a chance-adjusted index of agreement. In machine learning it can be used to quantify the amount of agreement between an algorithm's predictions and some trusted labels of the same objects. Kappa starts with accuracy - the proportion of all objects that both the algorithm and the trusted labels assigned to the same category or class. However, it then attempts to adjust for the probability of the algorithm and trusted labels assigning items to the same category "by chance." It does this by assuming that the algorithm and the trusted labels each have a predetermined quota for the proportion of objects to assign to each category. The original kappa coefficient assumed nominal categories but this was later extended to non-nominal categories through "weighting." The idea behind weighting is that some categories are more similar than others and thus some mismatching pairs of categories deserve varying degrees of "partial credit." Quadratic weights are one popular way of determining how much partial credit to assign to each mismatched pair of categories; there are other weights. I have more information about all of these concepts, including MATLAB functions, on my website: mreliability.jmgirard.com

See also: Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213–220.

Update: See my agreement package or Gwet's irrCAC package for R functions.

Answer 2

Here is a well explained example of Quadratic Weighted Kappa score: http://kagglesolutions.com/r/evaluation-metrics--quadratic-weighted-kappa