Quadratic weighted kappa versus linear weighted kappa

Question

When should I use quadratic weighted kappa or linear weighted kappa?

I have two observers evaluating the classes of a number of objects. The classes are fail, pass1, pass2, and excellent (ordinal scale). The errors in classification between "fail" or "excellent" and the different degrees of "pass" are more severe than errors between the classes of pass (pass1 and pass2).

Could I define the values of the classes as "fail = 1", "pass1= 20", "pass2=25", "excellent=40" and use linear weighted kappa (higher penalties on the extremes in a intuitive/subjective way)? Or should I use quadratic weighted kappa? Why should I prefer one kind over the other?

Answer 1

For four categories, the following linear and quadratic weights would be used. These tables can be read by indexing one rater by row and the other rater by column. For instance, raters would earn 0.33 "agreement credit" if one rater assigned the item to Pass2 (row 3) and the other assigned it to Fail (column 1). This is more than the 0.00 that would be award using nominal (i.e., identity) weights.

\begin{array} {|c|c|c|c|c|} \hline Linear& \text{Fail} & \text{Pass1} & \text{Pass2} & \text{Excel}\\ \hline \text{Fail} & 1.00 & 0.67 & 0.33 & 0.00 \\ \hline \text{Pass1} & 0.67 & 1.00 & 0.67 & 0.33 \\ \hline \text{Pass2} & 0.33 & 0.67 & 1.00 & 0.67 \\ \hline \text{Excel} & 0.00 & 0.33 & 0.67 & 1.00 \\ \hline \end{array}

\begin{array} {|c|c|c|c|c|} \hline Quadratic & \text{Fail} & \text{Pass1} & \text{Pass2} & \text{Excel}\\ \hline \text{Fail} & 1.00 & 0.89 & 0.56 & 0.00 \\ \hline \text{Pass1} & 0.89 & 1.00 & 0.89 & 0.56 \\ \hline \text{Pass2} & 0.56 & 0.89 & 1.00 & 0.89 \\ \hline \text{Excel} & 0.00 & 0.56 & 0.89 & 1.00 \\ \hline \end{array}

To choose between linear and quadratic weights, ask yourself if the difference between being off by 1 vs. 2 categories is the same as the difference between being off by 2 vs. 3 categories. With linear weights, the penalty is always the same (e.g., 0.33 credit is subtracted for each additional category). However, this is not the case for quadratic weights, where penalties begin mild then grow harsher.

\begin{array} {|c|c|c|c|} \hline \text{Difference} & \text{Linear} & \text{Quadratic} \\ \hline 0 & 1.00 & 1.00 \\ \hline 1 & 0.67 & 0.89 \\ \hline 2 & 0.33 & 0.56 \\ \hline 3 & 0.00 & 0.00 \\ \hline \end{array}

Also, in case anyone is interested, here are the formulas for both:

$$ \text{Linear: } w_i = 1 - \frac{i}{k-1} $$ $$ \text{Quadratic: } w_i = 1 - \frac{i^2}{(k-1)^2} $$ where $i$ is the difference between categories and $k$ is the total number of categories.