Wikipedia suggests that one way to look at inter-rater reliability is to use a random effects model to compute intraclass correlation. The example of intraclass correlation talks about looking at
$$\frac{\sigma_\alpha^2}{\sigma_\alpha^2+\sigma_\epsilon^2}$$
from a model
$$Y_{ij} = \mu + \alpha_i + \epsilon_{ij}$$
"where Yij is the jth observation in the ith group, μ is an unobserved overall mean, αi is an unobserved random effect shared by all values in group i, and εij is an unobserved noise term."
This is an attractive model especially because in my data no rater has rated all things (although most have rated 20+), and things are rated a variable number of times (usually 3-4).
Question #0: Is "group i" in that example ("group i") a grouping of things being rated?
Question #1: If I'm looking for inter-rater-reliability, don't I need a random effects model with two terms, one for the rater, and one for the thing rated? After all, both have possible variation.
Question #2: How would I best express this model in R?
It looks as if this question has a nice-looking proposal:
lmer(measurement ~ 1 + (1 | subject) + (1 | site), mydata)
I looked at a couple questions, and the syntax of the "random" parameter for lme is opaque to me. I read the help page for lme, but the description for "random" is incomprehensible to me without examples.
This question is somewhat similar to a long list of questions, with this the closest. However, most don't address R in detail.
