I have a question about one of the variables in my study and whether or not it should be considered a random effect.
I'm conducting a study of my school's 24 general learning outcomes (or "skills".) The 24 skills were divided up into 4 rubrics (6 skills each) with the goal of minimizing similarities between skills on the same rubric. For example, there are two skills relating to mathematics. One of them was assigned to rubric 1, and the other to rubric 2. We wanted to avoid putting both math skills on the same rubric.
We had 8 faculty raters who together assessed 72 samples of student work for all 24 skills over the course of two days. We'll call a single sample of student work a "selection". Each faculty member worked with a different single rubric on each of the two days (so 2 rubrics per rater in total, a different one each day). Thus, a single selection of student of work was rated for all 24 skills, but this work was divided among 4 different raters.
Looking at the correlation matrix, there is a high degree of correlation between skills on the same rubric, but we deliberately assigned skills to rubrics to try and avoid this. I am creating a mixed model to try and suss out why this correlation might have occurred. Is it because some raters grade more generously than others, or is it because faculty grade the student work "holistically", generally giving higher scores to one item on the rubric if they gave a high score to another item on the rubric? At the moment, my mixed model looks like this:
score ~ rater + rubric * skill + (1 | selection)
Here, score is the rating the sample received, rater is which faculty assigned the score, rubric is which of the 4 rubrics they used, skill is which of the skills they rated, and selection is the unique id for each sample of student work.
I've uploaded a small example of the data (with made-up values and only 3 rubrics with 2 skills each) in case that's helpful.
My question is, should the variable rater be a random effect?
Initially, I leaned toward making it a fixed effect, but reading the discussion here, prompted me to reconsider. In particular, several of the definitions of random effect provided by Gelman made me wonder. For example:
- Fixed effects are constant across individuals, and random effects vary. For example, in a growth study, a model with random intercepts αi and fixed slope β corresponds to parallel lines for different individuals i, or the model yit = αi + βt. Kreft and de Leeuw [(1998), page 12] thus distinguish between fixed and random coefficients.
The "individuals" in this study are selections of student work, and the rater varies across samples.
- Effects are fixed if they are interesting in themselves or random if there is interest in the underlying population. Searle, Casella and McCulloch [(1992), Section 1.4] explore this distinction in depth.
The variable rater is not interesting to me. In fact, it was not intended to have any effect on the scores. The fact that it may have an effect is more “nuisance” than “interesting” to me.
- “When a sample exhausts the population, the corresponding variable is fixed; when the sample is a small (i.e., negligible) part of the population the corresponding variable is random” [Green and Tukey (1960)].
The variable rater does not exhaust the population; there are several faculty who were not included in the study.
- “If an effect is assumed to be a realized value of a random variable, it is called a random effect” [LaMotte (1983)].
The raters were randomly assigned pools of student work so their effect on the scores should be random.
I know there is a lot of grey area when deciding which factors are fixed/random effects. I may be over-thinking this, but I want to make sure my model is correct so any guidance would be greatly appreciated.