3
$\begingroup$

My aim is to look at the reliability of a coding scheme which employs 7 exhaustive and mutually exclusive categories to categorise 40 "subjects" (i.e. a typical reliability study). More than 2 judges are being used (at the moment I've got 4, but will increase).

I want to look closer at the data to see if particular categories are being employed more reliably than others.

I am using R to analyse the data, and the "IRR" package allows "category-wise kappas" to be calculated, which reports the kappa for each individual category, however the percent agreement would be more useful to me. I'm also not sure how it calculates these numbers, which would be useful to know anyway (I know how kappas are calculated, just not the kappa for the specific codes).

$\endgroup$
3
$\begingroup$

The extended formula (for multiple categories and raters) for observed agreement is as follows: $$ A = \frac{1}{n'}\sum_{i=1}^{n'}\sum_{k=1}^{q}\frac{r_{ik}(r_{ik-1})}{r_i(r_i-1)} $$ where $n'$ is the number of items with two or more ratings, $q$ is the number of categories, $r_{ik}$ is the number of raters who assigned item $i$ to category $k$, and $r_i$ is the number of raters who assigned item $i$ to any category.

If you want to look at the specific agreement for a given category, it is:

$$ A_k = \frac{\sum_{i=1}^{n'}r_{ik}(r_{ik}-1)}{\sum_{i=1}^{n'}r_{ik}(r_i-1)} $$

The interpretation of this score is the probability of a randomly chosen rater assigning an item to that category given that another randomly chosen rater has also assigned that item to that category.

$\endgroup$
3
  • $\begingroup$ Hi thanks, you wouldn't happen to have a citation for the formula, or be able to provide a derivation of Ak. I understand that how rik(rik-1) and ri(ri-1) come from the combination of two formula but i don't quite understand where the denominator comes from in the Ak formula. I understand how to apply the formula, but would just be interested to know how its derived. $\endgroup$ Apr 11 '16 at 16:09
  • $\begingroup$ You can find more information about specific agreement at my website, including citations and the history of the measure. $\endgroup$ Apr 11 '16 at 16:21
  • $\begingroup$ Dear Jeffrey, I think this question is exactly related to your answer here? $\endgroup$
    – rnorouzian
    Oct 18 '19 at 1:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.