12 teachers are teaching 600 students. The 12 cohorts taught by these teachers range in size from 40 to 90 students, and we expect systematic differences between the cohorts, as graduate students were disproportionately allocated to particular cohorts, and previous experience has shown that the graduate students on average score considerably higher than the undergraduate students.

The teachers have graded all the papers in their cohort, and have assigned them a mark out of 100.

Each teacher has also looked at one randomly selected paper from three other teachers, and given it a mark out of 100. Each teacher has had three of his/her papers marked by another teacher. 36 different papers have thus been cross-marked in this way, and I call this my calibration data.

I can also see how many graduate students were in each cohort.

My questions are:

A) How can I use this calibration data to adjust the original marks in order to make them fairer? In particular, I'd like to wash out as much as possible the effects of overly generous/ungenerous makers.

B) How appropriate is my calibration data? I didn't have a choice in the rather limited 36 data points of calibration data I got in this course, and don't have any option to collect any more during the current semester. However, if this situation recurs I might be able to collect more calibration data or else collect different types of calibration data.

This question is a relative of a popular question I asked at: How can I best deal with the effects of markers with differing levels of generosity in grading student papers?. However, it's a different course and I'm not sure how useful reading that question would be as background for this current one, since the chief problem there was that I had no calibration data.


2 Answers 2


This sounds like a great opportunity to use a matrix factorization recommender system. Briefly, this works as follows:

  • Put your observations into a partially-observed matrix $M$ where $M_{ij}$ is the score teacher $i$ gave to student $j$.

  • Assume that this matrix is the outer product of some latent feature vectors, $\vec t$ and $\vec s$--that is, $M_{ij} = t_i s_j$.

  • Solve for the latent feature vectors that minimize the squared reconstruction error $\sum_{i,j} (t_is_j - M_{ij})^2$ (where the sum ranges over all observed cells of $M$).

  • You can do this expectation-maximization style by fixing a guess for $\vec t$ and solving for $\vec s$ via least squares, then fixing that guess for $\vec s$ and solving for $\vec t$ and iterating until convergence.

Note that this makes a fairly strong assumption on the form of a teacher's bias--in particular, if you think of the students' latent features as their "true score", then a teacher's bias multiplies each true score by a constant amount (to make it additive instead you would exponentiate the scores that you insert into the matrix, and then learn the exponentials of the "true scores"). With so little calibration data, you probably can't get very far without making a strong assumption of this form, but if you had more data, you could add a second dimension of latent features, etc. (i.e., assume $M_{ij} = \sum_{k=1}^n s_{ik} t_{kj}$ and again try to minimize the squared reconstruction error).

EDIT: in order to have a well-defined problem you need to have more matrix operations than latent parameters (or you can use some kind of regularization). You just barely have that here (you have 636 observations and 612 latent parameters), so the matrix factorization may not work super well--I haven't worked with them on such small samples, so I don't really know.

If the calibration turns out to be insufficient to use a good recommender model, you could try a multilevel regression on Score ~ IsGradStudent + <whatever other student covariates you have> + (1|Teacher) (ignoring the calibration data) to extract estimates of an additive teacher bias, and then check if this bias is consistent with the calibration data you took. (You should allow for heteroskedasticity by teacher if possible.) This is more ad-hoc but may give you less severe data collection problems.

  • $\begingroup$ To expand on this, I'd probably start with a simple model with teacher fixed effects and potentially clustered robust standard errors (see this blog post for a discussion of this in R) and then compare the fixed effects for any outliers. In R, something like lm(score ~ gradStudent + ... + teacherID should do it. $\endgroup$
    – iacobus
    Commented Apr 16, 2015 at 22:02

Here's a couple of related approaches.

Take the set of papers marked by more than one teacher, since those contain the most information about teacher effects and outside those papers, the teacher and cohort effects are confounded (if there was some way of getting at the cohort effect -- perhaps via GPA or some other predictor, for example, then you could use all of the data, but it will complicate the models quite a bit).

Label the students $i=1,2, ... n$, and the markers $j=1, 2, ...,m$. Let the set of marks be $y_{ij}, i=1,2, ... m$.

You first have to consider your model for how the marker-effect applies. Is it additive? Is it multiplicative? Do you need to worry about boundary effects (e.g. would an additive or multiplicative effect on a logit-scale be better)?

Imagine two given markers on two papers and imagine the second marker is more generous. Let's say the first marker would give the papers 30 and 60. Will the second marker tend to add a constant number of marks (say 6 marks) to both? Will they tend to add constant percentages (say 10% to both, or 3 marks vs 6 marks)? What if the first marker gave 99? -- what would happen then? What about 0? What if the second marker were less generous? what would happen at 99 or 0? (this is why I mention a logit model - one might treat the marks as a proportion of the possible marks ($p_{ij}=m_{ij}/100$), and then the marker effect could be to add a constant (say) to the logit of $p$ - i.e. $\log(p_{ij}/(1-p_{ij})$).

(You won't have enough data here to estimate the form of generousness as well as its size. You have to choose a model from your understanding of the situation. You'll also need to ignore any possibility of interaction; you don't have the data for it)

Possibility 1 - plain additive model. This might be suitable if no marks were really close to 0 or 100:

Consider a model like $E(y_{ij}) = \mu_{i}+\tau_j$

This is essentially a two-way ANOVA. You need constraints on this, so you might set up a deviation coding/set up the model so that of marker effects is 0, or you might set up a model where one marker is the baseline (whose effect is 0, and whose marks you will try to adjust every other marker toward).

Then take the $\hat{\tau}_j$ values and adjust the wider population of marks $y_{kj}^\text{adj}=y_{kj}-\hat{\tau}_j$.

Possibility 2: In effect, a similar kind of idea but $E(y_{ij}) = \mu_{i}\tau_j$. Here you might fit a nonlinear least squares model, or a GLM with a log-link (I'd probably lean toward the second out of those two). Again you need a constraint on the $\tau$s.

Then a suitable adjustment would be to divide by $\hat{\tau_j}$.

Possibility 3: additive on the logit scale. This might be more suitable if some marks get close to 0 or 100. It will look roughly multiplicative for very small marks, additive for middling marks and roughly multiplicative in $1-p=(100-m)/100$ for very high marks. You might use a beta regression or a quasi-binomial GLM with logit link to fit this model.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.