My aim is to look at the reliability of a coding scheme which employs 7 exhaustive and mutually exclusive categories to categorise 40 "subjects" (i.e. a typical reliability study). More than 2 judges are being used (at the moment I've got 4, but will increase).
I want to look closer at the data to see if particular categories are being employed more reliably than others.
I am using R to analyse the data, and the "IRR" package allows "category-wise kappas" to be calculated, which reports the kappa for each individual category, however the percent agreement would be more useful to me. I'm also not sure how it calculates these numbers, which would be useful to know anyway (I know how kappas are calculated, just not the kappa for the specific codes).