# Computing inter-rater reliability in R with variable number of ratings?

Wikipedia suggests that one way to look at inter-rater reliability is to use a random effects model to compute intraclass correlation. The example of intraclass correlation talks about looking at

$$\frac{\sigma_\alpha^2}{\sigma_\alpha^2+\sigma_\epsilon^2}$$

from a model

$$Y_{ij} = \mu + \alpha_i + \epsilon_{ij}$$

"where Yij is the jth observation in the ith group, μ is an unobserved overall mean, αi is an unobserved random effect shared by all values in group i, and εij is an unobserved noise term."

This is an attractive model especially because in my data no rater has rated all things (although most have rated 20+), and things are rated a variable number of times (usually 3-4).

Question #0: Is "group i" in that example ("group i") a grouping of things being rated?

Question #1: If I'm looking for inter-rater-reliability, don't I need a random effects model with two terms, one for the rater, and one for the thing rated? After all, both have possible variation.

Question #2: How would I best express this model in R?

It looks as if this question has a nice-looking proposal:

lmer(measurement ~ 1 + (1 | subject) + (1 | site), mydata)


I looked at a couple questions, and the syntax of the "random" parameter for lme is opaque to me. I read the help page for lme, but the description for "random" is incomprehensible to me without examples.

This question is somewhat similar to a long list of questions, with this the closest. However, most don't address R in detail.

The model you referenced in your question is called the "one-way model." It assumes that random row effects are the only systematic source of variance. In the case of inter-rater reliability, rows correspond to objects of measurement (e.g., subjects).

One-way model: $$x_{ij} = \mu + r_i + w_{ij}$$ where $\mu$ is the mean for all objects, $r_i$ is the row effect, and $w_{ij}$ is the residual effect.

However, there are also "two-way models." These assume that there is variance associated with random row effects as well as random or fixed column effects. In the case of inter-rater reliability, columns correspond to sources of measurement (e.g., raters).

Two-way models: $$x_{ij} = \mu + r_i + c_j + rc_{ij} + e_{ij}$$$$x_{ij} = \mu + r_i + c_j + e_{ij}$$ where $\mu$ is the mean for all objects, $r_i$ is the row effect, $c_j$ is the column effect, $rc_{ij}$ is the interaction effect, and $e_{ij}$ is the residual effect. The difference between these two models is the inclusion or exclusion of the interaction effect.

Given a two-way model, you can calculate one of four ICC coefficients: the single score consistency ICC(C,1), the average score consistency ICC(C,k), the single score agreement ICC(A,1), or the average score agreement ICC(A,k). Single score ICCs apply to single measurements $x_{ij}$ (e.g., individual raters), whereas average score ICCs apply to average measurements $\bar{x}_i$ (e.g., the mean of all raters). Consistency ICCs exclude the column variance from the denominator variance (e.g., allowing raters to vary around their own means), whereas agreement ICCs include the column variance in the denominator variance (e.g., requiring raters to vary around the same mean).

Here are the definitions if you assume a random column effect:

Two-way Random-Effects ICC Definitions (with or without interaction effect): $$ICC(C,1) = \frac{\sigma_r^2}{\sigma_r^2 + (\sigma_{rc}^2 + \sigma_e^2)}\text{ or }\frac{\sigma_r^2}{\sigma_r^2 + \sigma_e^2}$$ $$ICC(C,k) = \frac{\sigma_r^2}{\sigma_r^2 + (\sigma_{rc}^2 + \sigma_e^2)/k}\text{ or }\frac{\sigma_r^2}{\sigma_r^2 + \sigma_e^2/k}$$ $$ICC(A,1) = \frac{\sigma_r^2}{\sigma_r^2 + (\sigma_c^2 + \sigma_{rc}^2 + \sigma_e^2)}\text{ or }\frac{\sigma_r^2}{\sigma_r^2 + (\sigma_c^2 + \sigma_e^2)}$$ $$ICC(A,k) = \frac{\sigma_r^2}{\sigma_r^2 + (\sigma_c^2 + \sigma_{rc}^2 + \sigma_e^2)/k}\text{ or }\frac{\sigma_r^2}{\sigma_r^2 + (\sigma_c^2 + \sigma_e^2)/k}$$

You can also estimate these values using mean squares from ANOVA:

Two-way ICC Estimations: $$ICC(C,1) = \frac{MS_R - MS_E}{MS_R + (k-1)MS_E}$$ $$ICC(C,k) = \frac{MS_R-MS_E}{MS_R}$$ $$ICC(A,1) = \frac{MS_R-MS_E}{MS_R + (k-1)MS_E + k/n(MS_C-MS_E)}$$ $$ICC(A,k) = \frac{MS_R-MS_E}{MS_R + (MS_C-MS_E)/n}$$

You can calculate these coefficients in R using the irr package:

icc(ratings, model = c("oneway", "twoway"),
type = c("consistency", "agreement"),
unit = c("single", "average"), r0 = 0, conf.level = 0.95)


References

McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46.

Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428.

• Thanks for the great answer! In a two-way model within icc in R, how do we represent random selection of raters per row? I mean, imagine we have a pool of 100 raters, and each subject is rated by around 5-10 of them. Can such a scenario be handled by icc package? Sep 21, 2017 at 11:10
• Each rater should have their own column in the matrix you feed to the icc function. Otherwise, the computation is the same for random and mixed effects models - the main difference is in interpretation (how generalizable the results can be considered). Sep 21, 2017 at 12:05
• Thanks for the answer! I am trying to do that, having mostly NA in the cells (and only a few values with actual numbers per column, where a particular rater rated a subject corresponding to a row). However, in the output I am getting a text saying no subjects were recorded (e.g. Subjects = 0 Raters = 9 ). Perhaps it means that wherever at least one NA was found the whole row is filtered out? But then how can i denote missing ratings from a rater? Sep 21, 2017 at 12:13
• Hmm that may be a limitation of this specific icc function. I have a MATLAB script that can handle this situation. Do you happen to have access to MATLAB? Sep 21, 2017 at 12:24
• Yeah, check out my website: mreliability.jmgirard.com Sep 21, 2017 at 13:14