Consider the following slightly unrealistic experiment:

I have an infinitely large pool of questions to write exams from, and suppose that the questions are continuously rated in difficulty from 0 to 10 and that each difficulty rating has an infinite number of questions in the pool corresponding to it. The exams are administered by a computer.

When a student goes to his computer he is randomly (uniformly) assigned how much the difficulty from question to question is going to vary, $\delta$, and an exam is then generated for him by drawing normally distributed questions (in difficulty) with mean difficulty level 5 and variance in difficulty level $\delta$. The student after finishing the exam and receiving their score then chooses whether they would like to take another exam. If they do, another exam is generated for them with the same variance $\delta$.

So a single student has the same $\delta$ for all of his exams, but any two students will have two different $\delta$s. I imagine two interesting questions I could ask from this data. (1) What is the effect of $\delta$ on exam score, and (2) what is the effect of the number of exams the student has taken so far on his current exam score (i.e. how much do the students learn?). However, if each student gets the same $\delta$ for all his exams, but different $\delta$s across students, it seems difficult to tease apart the effect of student ability (which I don't care about - and is presumably random) with the effect of $\delta$.

What kind of model should I use to answer the above questions (1) and (2)? My first instinct is perhaps a mixed effect model where $\delta$ is a fixed effect, the number of exams taken at the time of each exam is also a fixed effect, and each student is a random effect. Note, I find mixed effect models very confusing, so perhaps the above statement made no sense.

I imagine student effect might be important because students who perform a certain way might choose to take more exams while students who perform another way might choose to stop taking exams earlier. For example if poorly performing students choose to take more exams to try and improve their score and good students are satisfied with their first exam a preliminary look at the data might conclude that students do worse after several exams, without a student effect built in.

data would be of the form:

  • student 1: $\delta=2.11$, exam 1 = 28, exam 2 =40, exam 3 = 39, exams taken =3
  • student 2: $\delta=0.23$, exam 1 = 44, exams taken =1
  • student 3: $\delta=1.83$, exam 1 = 21, exam 2 = 9, exam 3 = 18, exam 4 = 24, exams taken =4
  • .
  • .
  • .
  • student N: $\delta=1.24$, exam 1 = 51, exam 2 = 56, exams taken =2
  • 1
    $\begingroup$ Without going into too much detail, this sounds similar to item response theory, which can be handled via crossed random effects in lme4. There is a whole section on item response theory in the Psychometrics task view. I would include the number of exams taken to date as a continuous covariate. $\endgroup$ – Ben Bolker Dec 28 '13 at 23:01
  • $\begingroup$ (On second thought, maybe not: in IRT we assume we have data on each question ...) $\endgroup$ – Ben Bolker Dec 28 '13 at 23:03
  • 1
    $\begingroup$ One more thought: a mixed model with a random effect of student, fixed effects of $\delta$ and cumulative number of exams taken, and a variance effect of $\delta$ (in nlme: using weights=varFixed(~delta) $\endgroup$ – Ben Bolker Dec 28 '13 at 23:17
  • $\begingroup$ Thanks Ben, I finally figured it all out (although I do wish the nlme documentation was as thoroughly written as lme4, which is really good), and your paper in trends and evolution cleared up a lot of general questions I had about GLMMs and LMEs. Great to have something out there for beginners like myself. Thanks again! $\endgroup$ – WetlabStudent Jan 10 '14 at 0:00
  • $\begingroup$ well, to be fair there is a whole, well-written (in my opinion) book (Pinheiro and Bates, Springer 2000) covering nlme ... $\endgroup$ – Ben Bolker Jan 10 '14 at 3:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.