0
$\begingroup$

We have 2 different interpretations for our data. Here is an example:

Interpretation-1    Interpretation-2
Frame   tool1    Frame  tool1
1       0        1      1
2       1        2      2
3       1        3      2
4       1        4      3
5       2        5      3
6       2        6      3
7       0        7      1
8       0        8      0
..               ..

The tool1 interpretation represents the number of time this tool exists in the frame.

I know how to compute the Cohen's kappa for a binary classification problem but I didn't find any useful tutorial to compute it in such case.

Improve this question
$\endgroup$
2
  • $\begingroup$ I take it you want to compare interpretation-1's tool1 measure and interpretation-2's tool1 measure, and these measures are both counts (i.e., ratio scales)? $\endgroup$
    – Jeffrey Girard
    Commented Apr 9, 2017 at 0:07
  • $\begingroup$ yep that's it.. $\endgroup$
    – Maystro
    Commented Apr 10, 2017 at 7:38

1 Answer 1

Reset to default
1
$\begingroup$

You can use a weighted version of kappa (or other chance-adjusted agreement index). Since you have ratio data, you should use a "ratio" or "count" weighting scheme. Basically, make a new matrix that has each row corresponding to a single frame and each column corresponding to an interpretation (i.e., remove the first and third column in your example table, as well as the header rows). Then you can input this matrix to a function that calculates weighted kappa. I provide the formulas and didactic information on my mReliability website, as well as MATLAB functions. Kilem Gwet has an excellent textbook on this topic and also provides functions for Excel, SAS, and R on his AgreeStat website.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.