# How can I calculate percent agreement and kappa for each category?

My aim is to look at the reliability of a coding scheme which employs 7 exhaustive and mutually exclusive categories to categorise 40 "subjects" (i.e. a typical reliability study). More than 2 judges are being used (at the moment I've got 4, but will increase).

I want to look closer at the data to see if particular categories are being employed more reliably than others.

I am using R to analyse the data, and the "IRR" package allows "category-wise kappas" to be calculated, which reports the kappa for each individual category, however the percent agreement would be more useful to me. I'm also not sure how it calculates these numbers, which would be useful to know anyway (I know how kappas are calculated, just not the kappa for the specific codes).

The extended formula (for multiple categories and raters) for observed agreement is as follows: $$A = \frac{1}{n'}\sum_{i=1}^{n'}\sum_{k=1}^{q}\frac{r_{ik}(r_{ik-1})}{r_i(r_i-1)}$$ where $n'$ is the number of items with two or more ratings, $q$ is the number of categories, $r_{ik}$ is the number of raters who assigned item $i$ to category $k$, and $r_i$ is the number of raters who assigned item $i$ to any category.
$$A_k = \frac{\sum_{i=1}^{n'}r_{ik}(r_{ik}-1)}{\sum_{i=1}^{n'}r_{ik}(r_i-1)}$$