0
$\begingroup$

I have some trouble calculating the inter-rater reliability for a nominal (0/1) variable. The study design was as follows: different subjects were rated by different subsets of coders, in which each subject was rated by two coders. Lately, I wanted to make a start with assessing the IRR using Kappa Statistics. To do so, I structured the data with separate variables for each coder (see the print screen below). However, at the moment I wanted to calculate the Kappa Statistic, I discovered that a Kappa Statistics can only be calculated when all the ratings from the different coders are merged in two columns (i.e. one column for rater 1 and one column for rater 2). Since each subject is rated by a different set of raters, my dataset contains more than 2 columns, which hinders me to calculate a Kappa Statistic for the variable of interest. I searched for a solution, but unfortunately couldn’t find one. My basic question therefore is: how to calculate a Kappa Statistic in a dataset like mine? Would it make sense to just merge all the ratings in two columns or is there a better solution?

Any advice will be highly appreciated!

Print screen dataset

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

The original kappa formula (and that used by SPSS) converts the data into a $q$-by-$q$ confusion matrix (item counts in a category-by-category matrix). This formula required exactly two raters. Later "extended" formulas convert the data into an $n$-by-$q$ classification matrix (rater counts in an item-by-category matrix). This formulation allowed any number of raters and various patterns of missingness. The intuition here is that with $X$ raters rating a given item, there are $X(X-1)$ possible agreements.

Click here to see more information and MATLAB functions.

Uebersax, J. (1982). A design-independent method for measuring the reliability of psychiatric diagnosis. Journal of Psychiatric Research. 17(4). 335-342.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ In case you don't know, there are math typesetting tools on this site that, so that for instance $x$ renders as $x$. There are some editing tips here. $\endgroup$
    – Silverfish
    Apr 9, 2016 at 14:19
  • $\begingroup$ @Silverfish Thanks for the edits. I will be more careful about using inline math in the future. $\endgroup$
    – Jeffrey Girard
    Apr 9, 2016 at 14:28
  • $\begingroup$ It's not a big deal, I do think it is easier to read but I mostly do it because it looks prettier! $\endgroup$
    – Silverfish
    Apr 9, 2016 at 14:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.