Consider the following slightly unrealistic experiment:
I have an infinitely large pool of questions to write exams from, and suppose that the questions are continuously rated in difficulty from 0 to 10 and that each difficulty rating has an infinite number of questions in the pool corresponding to it. The exams are administered by a computer.
When a student goes to his computer he is randomly (uniformly) assigned how much the difficulty from question to question is going to vary, $\delta$, and an exam is then generated for him by drawing normally distributed questions (in difficulty) with mean difficulty level 5 and variance in difficulty level $\delta$. The student after finishing the exam and receiving their score then chooses whether they would like to take another exam. If they do, another exam is generated for them with the same variance $\delta$.
So a single student has the same $\delta$ for all of his exams, but any two students will have two different $\delta$s. I imagine two interesting questions I could ask from this data. (1) What is the effect of $\delta$ on exam score, and (2) what is the effect of the number of exams the student has taken so far on his current exam score (i.e. how much do the students learn?). However, if each student gets the same $\delta$ for all his exams, but different $\delta$s across students, it seems difficult to tease apart the effect of student ability (which I don't care about - and is presumably random) with the effect of $\delta$.
What kind of model should I use to answer the above questions (1) and (2)? My first instinct is perhaps a mixed effect model where $\delta$ is a fixed effect, the number of exams taken at the time of each exam is also a fixed effect, and each student is a random effect. Note, I find mixed effect models very confusing, so perhaps the above statement made no sense.
I imagine student effect might be important because students who perform a certain way might choose to take more exams while students who perform another way might choose to stop taking exams earlier. For example if poorly performing students choose to take more exams to try and improve their score and good students are satisfied with their first exam a preliminary look at the data might conclude that students do worse after several exams, without a student effect built in.
data would be of the form:
- student 1: $\delta=2.11$, exam 1 = 28, exam 2 =40, exam 3 = 39, exams taken =3
- student 2: $\delta=0.23$, exam 1 = 44, exams taken =1
- student 3: $\delta=1.83$, exam 1 = 21, exam 2 = 9, exam 3 = 18, exam 4 = 24, exams taken =4
- student N: $\delta=1.24$, exam 1 = 51, exam 2 = 56, exams taken =2