# which statistic should I use for inter-rater agreement on nominal data, multiple responses

I have over 400 open-ended responses. The total has been coded by one rater, and a 10% identical sample of the total has been coded by 2 raters. There is a choice of 14 nominal codes, of which the rater can use any number per case, though most cases have between 1-4 codes. Which statistical test is the most appropriate to determine if I have inter-rater agreement on the 10% sample?

Also am I right in assuming that I will need to treat each code as a variable and look for the level of agreement for each variable?

I'm using SPSS 21.

Much thanks, Kellie

You could use a chance-adjusted agreement index (e.g., Cohen's kappa or Scott's pi) for each category separately. Alternatively, you could use the following approach:

Kramer (1980) proposed a method for assessing inter-rater reliability for tasks in which raters could select multiple categories for each object of measurement. The intuition behind this method is to reframe the problem from one of classification to one of rank ordering. Thus, all selected categories are tied for first place and all non-selected categories are tied for second place. Chance-adjusted agreement can then be calculated using rank correlation coefficients or analysis of variance of the ranks. Naturally, this approach also allows multiple categories to be ranked by raters.

$$\kappa_0 = \frac{\bar{P} - P_e}{1 - P_e} + \frac{1 - \bar{P}}{Nm_0(1 - P_e)}$$ where $\bar{P}$ is the average proportion of concordant pairs out of all possible pairs of observations for each subject, $P_e=\sum_j p_j^2$ and $p_j$ is the overall proportion of observations in which response category $j$ was selected, $m_0$ is the number of observations per subject, and $N$ is the number of subjects. It can also be shown that, when only one category is selected, $\kappa_0$ asymptotically approaches Cohen's and Fleiss' kappa coefficients.

A clever solution, but not one that I've ever seen used in an article.

References

Kraemer, H. C. (1980). Extension of the kappa coefficient. Biometrics, 36(2), 207–16.

• In those situations, how common is it to use a chance-adjusted agreement index for each category separately?
– DomB
Aug 28, 2020 at 17:30