0
$\begingroup$

We have 2 different interpretations for our data. Here is an example:

Interpretation-1    Interpretation-2
Frame   tool1    Frame  tool1
1       0        1      1
2       1        2      2
3       1        3      2
4       1        4      3
5       2        5      3
6       2        6      3
7       0        7      1
8       0        8      0
..               ..

The tool1 interpretation represents the number of time this tool exists in the frame.

I know how to compute the Cohen's kappa for a binary classification problem but I didn't find any useful tutorial to compute it in such case.

$\endgroup$
  • $\begingroup$ I take it you want to compare interpretation-1's tool1 measure and interpretation-2's tool1 measure, and these measures are both counts (i.e., ratio scales)? $\endgroup$ – Jeffrey Girard Apr 9 '17 at 0:07
  • $\begingroup$ yep that's it.. $\endgroup$ – Maystro Apr 10 '17 at 7:38
1
$\begingroup$

You can use a weighted version of kappa (or other chance-adjusted agreement index). Since you have ratio data, you should use a "ratio" or "count" weighting scheme. Basically, make a new matrix that has each row corresponding to a single frame and each column corresponding to an interpretation (i.e., remove the first and third column in your example table, as well as the header rows). Then you can input this matrix to a function that calculates weighted kappa. I provide the formulas and didactic information on my mReliability website, as well as MATLAB functions. Kilem Gwet has an excellent textbook on this topic and also provides functions for Excel, SAS, and R on his AgreeStat website.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.