4
$\begingroup$

I have a question about one of the variables in my study and whether or not it should be considered a random effect.

I'm conducting a study of my school's 24 general learning outcomes (or "skills".) The 24 skills were divided up into 4 rubrics (6 skills each) with the goal of minimizing similarities between skills on the same rubric. For example, there are two skills relating to mathematics. One of them was assigned to rubric 1, and the other to rubric 2. We wanted to avoid putting both math skills on the same rubric.

We had 8 faculty raters who together assessed 72 samples of student work for all 24 skills over the course of two days. We'll call a single sample of student work a "selection". Each faculty member worked with a different single rubric on each of the two days (so 2 rubrics per rater in total, a different one each day). Thus, a single selection of student of work was rated for all 24 skills, but this work was divided among 4 different raters.

Looking at the correlation matrix, there is a high degree of correlation between skills on the same rubric, but we deliberately assigned skills to rubrics to try and avoid this. I am creating a mixed model to try and suss out why this correlation might have occurred. Is it because some raters grade more generously than others, or is it because faculty grade the student work "holistically", generally giving higher scores to one item on the rubric if they gave a high score to another item on the rubric? At the moment, my mixed model looks like this:

score ~ rater + rubric * skill + (1 | selection)

Here, score is the rating the sample received, rater is which faculty assigned the score, rubric is which of the 4 rubrics they used, skill is which of the skills they rated, and selection is the unique id for each sample of student work.

I've uploaded a small example of the data (with made-up values and only 3 rubrics with 2 skills each) in case that's helpful.

My question is, should the variable rater be a random effect?

Initially, I leaned toward making it a fixed effect, but reading the discussion here, prompted me to reconsider. In particular, several of the definitions of random effect provided by Gelman made me wonder. For example:

  1. Fixed effects are constant across individuals, and random effects vary. For example, in a growth study, a model with random intercepts αi and fixed slope β corresponds to parallel lines for different individuals i, or the model yit = αi + βt. Kreft and de Leeuw [(1998), page 12] thus distinguish between fixed and random coefficients.

The "individuals" in this study are selections of student work, and the rater varies across samples.

  1. Effects are fixed if they are interesting in themselves or random if there is interest in the underlying population. Searle, Casella and McCulloch [(1992), Section 1.4] explore this distinction in depth.

The variable rater is not interesting to me. In fact, it was not intended to have any effect on the scores. The fact that it may have an effect is more “nuisance” than “interesting” to me.

  1. “When a sample exhausts the population, the corresponding variable is fixed; when the sample is a small (i.e., negligible) part of the population the corresponding variable is random” [Green and Tukey (1960)].

The variable rater does not exhaust the population; there are several faculty who were not included in the study.

  1. “If an effect is assumed to be a realized value of a random variable, it is called a random effect” [LaMotte (1983)].

The raters were randomly assigned pools of student work so their effect on the scores should be random.

I know there is a lot of grey area when deciding which factors are fixed/random effects. I may be over-thinking this, but I want to make sure my model is correct so any guidance would be greatly appreciated.

$\endgroup$
  • 1
    $\begingroup$ I would lean towards treating it as a random effect, but it's a grey area for at least one reason: it's frequently recommended to have many more levels than just 8 when using them. $\endgroup$ – mkt Aug 18 at 16:22
  • 1
    $\begingroup$ Do both and check if it makes a difference by carrying out the Hausman test (the null hypothesis is that the random effects model is appropriate). $\endgroup$ – CloseToC Aug 18 at 17:05
  • 2
    $\begingroup$ If you can assume raters were selected at random from a population of qualified people, then raters should be considered as a random effect. Particularly, if this assessment will be done periodically with the exactly the same panel of raters each time, then raters should be considered as a fixed effect. // Suppose I have a business that makes plastic parts using five machines. I do a study trying different brands of plastic powder on each machine. Machines are a fixed effect. A salesman asks for my data. He may consider my 5 machines to have been selected at random, and treat them as random. $\endgroup$ – BruceET Aug 18 at 18:08
  • $\begingroup$ @mkt I chose to make rubric a fixed effect precisely because of the small number of levels. Ben Bolker's FAQ suggests that " for practical purposes, there must be a reasonable number of random-effects levels (e.g. blocks) – more than 5 or 6 at a minimum." The number of raters, 8 is above that minimum so I was thinking rater could still qualify as a random effect. Admittedly though, the small number of levels is not ideal. $\endgroup$ – forest-fisher Aug 18 at 18:23
  • 1
    $\begingroup$ @BruceET Raters were self-selected from a larger population of qualified people and compensated for their work. Assessment is done periodically with a rotating cast of roughly 30-40 people, but this particular assessment was unique in that we were trying to gather data on the rubrics themselves and the relationship between skills described on the rubrics. That is, in this assessment, the rubrics were the object of study, whereas normally student learning is the object of study (and the rubrics are a tool to help us understand that learning.) $\endgroup$ – forest-fisher Aug 18 at 18:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.