# How can I best deal with the effects of markers with differing levels of generosity in grading student papers?

Around 600 students have a score on an extensive piece of assessment, which can be assumed to have good reliability/validity. The assessment is scored out of 100, and it's a multiple-choice test marked by computer.

Those 600 students also have scores on a second, minor, piece of assessment. In this second piece of assessment they are separated into 11 cohorts with 11 different graders, and there is an undesirably large degree of variation between graders in terms of their 'generosity' in marking, or lack thereof. This second assessment is also scored out of 100.

Students were not assigned to cohorts randomly, and there are good reasons to expect differences in skill levels between cohorts.

I'm presented with the task of ensuring that differences between cohort markers on the second assignment don't materially advantage/disadvantage individual students.

My idea is to get the cohort scores on the second assessment to cohere with cohort scores on the first, while maintaining individual differences within the cohorts. We should assume that I have good reasons to believe that performance on the two tasks will be highly correlated, but that the markers differ considerably in their generosity.

Is this the best approach? If not, what is?

It'd be greatly appreciated if the answerer could give some practical tips about how to implement a good solution, say in R or SPSS or Excel.

• Great question! Are the final scores for the multiple choice and the essay portions supposed to be comparable (ie the same numerical ranges)? – gung - Reinstate Monica Nov 13 '14 at 3:33
• As I was writing this question I thought it might be up your alley! The final scores are broadly comparable, but a bit different. The mean on the multiple choice section is ~70 with a SD around 15. The mean on the other section is ~85 with a SD around 6. – user1205901 - Reinstate Monica Nov 13 '14 at 3:37
• I would be suspicious of any effort to solve this problem based only on the data you have described, because it would have to rest on the strong (and untestable) assumption that there is no interaction between cohort and performance on the two separate test instruments. If you possibly can, consider the option of conducting a separate small experiment to calibrate the graders. – whuber Nov 13 '14 at 4:14
• To see better where the problem lies, suppose (hypothetically) that (1) the two forms of assessment are multiple choice and essay and (2) your older students tend to do relatively better on essay questions. When you use your data to make the scores "cohere" you will be confounding the grader effects with the age effects and, by making adjustments, thereby systematically disadvantage the older students compared to the younger. No matter how sophisticated an algorithm you choose, it can only paper over this basic problem. You need some additional data to resolve this confounding. – whuber Nov 13 '14 at 4:44
• One thing to consider is how comfortable you'd be explaining the adjustment procedure to students or other stakeholders: many might feel that given a potential issue with the marking, putting some effort into a proper calibration of markers would not be too much to expect if the exam's an important one. – Scortchi - Reinstate Monica Nov 17 '14 at 13:40

Suppose students A, B, C and D, all in the same cohort, "should" be graded as 75, 80, 85 and 90 respectively but their generous grader consistently marks 5 marks too high. We observe 80, 85, 90 and 95 and should subtract 5, but finding the figure to subtract is problematic. It can't be done by comparing results between cohorts since we expect cohorts to vary in average ability. One possibility is using the multiple choice test results to predict the correct scores on the second assignment, then use this to assess variation between each grader and the correct grades. But making this prediction is non-trivial - if you expect different mean and standard deviation between the two assessments, you can't just assume that the second assessment grades should match the first.

Also, students differ in relative aptitude at multiple-choice and written assessments. You could treat that as some kind of random effect, forming a component of the student's "observed" and "true" grades, but not captured by their "predicted" grade. If cohorts differ systematically and students in a cohort tend be similar, then we shouldn't expect this effect to average out to zero within each cohort. If a cohort's observed grades average +5 versus their predicted ones, it is impossible to determine whether this is due to a generous grader, a cohort particularly better-suited to written assessment than multiple-choice, or some combination of the two. In an extreme case, the cohort may even have lower aptitude at the second assessment but had this more than compensated for by a very generous grader - or vice versa. You can't break this apart. It's confounded.

Your graders behave very differently. Are you sure they differ only in their overall generosity, rather than in their generosity in various components of the assessment? This might be worth checking, as it could introduce various complications - e.g. the observed grade for B may be worse than that of A, despite B being 5 point "better", even if the grader's allocated marks for each component are a monotonically increasing function of the Lead Assessor's! Suppose the assessment is split between Q1 (A should score 30/50, B 45/50) and Q2 (A should score 45/50, B 35/50). Imagine the grader is very lenient on Q1 (observed grades: A 40/50, B 50/50) but harsh on Q2 (observed: A 42/50, 30/50), then we observe totals of 82 for A and 80 for B. If you do have to consider component scores, note that clipping may be an issue - I suspect few papers get graded a perfect 100, but rather more papers will be awarded full marks in at least one component.

• Unfortunately the nature of assessment 2 makes it impossible for the graders to look at more for calibration purposes. You can think of it as being like an oral poetry recitation that was done once with no recording, and which was assessed immediately afterwards. It would be impractical to schedule new recitations purely for calibration purposes. To answer your other question, Assessment 2 didn't really have clear subcomponents, and we don't need to consider component scores. – user1205901 - Reinstate Monica Nov 18 '14 at 10:32
• This is "not an answer" but in an ideal world I'd have suggested to turn things around and use an example sample (possibly of artificial assignments deliberately designed to be on grade borderlines, rather than by real students) as a way of training the graders to have the same generosity, rather than to deduce and compensate for their generosities. If the assessments are done this is clearly no solution for you, though. – Silverfish Nov 18 '14 at 11:23
• (+1) Very thorough "not an answer". Consistency in rather subjective tests can often be greatly improved by splitting the grading task into components - otherwise one grader might be giving more weight to rhythm, another to projection, &c. – Scortchi - Reinstate Monica Nov 18 '14 at 11:36
• It is clear that in addition to submitting a possible adjustment to the person who will ultimately decide the issue, I will also need to submit some explanation of the pros and cons of adjustment. Your response provides a lot of helpful material regarding this. However, I wonder what criteria I can use to make a judgement on whether it's more beneficial to leave everything alone, or to make a change. I look at the cohort grades and my intuition says that the differences between markers are a having a big impact. Intuition is unreliable, but I'm not sure what else I can go on in this case. – user1205901 - Reinstate Monica Nov 18 '14 at 11:47
• One question is whether you have reasonable grounds to believe the "differential task aptitude" effect to be small, particularly when averaged over a cohort, compared to the "grader generosity" effect. If so, you might attempt to to estimate the generosity effect for each cohort - but you risk being confounded. Moreover, there is a Catch 22. I would be most cautious of applying large "corrections" to the observed grades. But if suggested corrections are small, it is plausible they are due to systematic differences in differential task ability between cohorts, not grader generosity at all. – Silverfish Nov 18 '14 at 13:19

A very simple model:

Let $s_{1,i}$ be the score of student $i$ on test 1, and $s_{2,i}$ his score on test 2. Let $A_1, \ldots, A_p$ be the partition of the students in the original cohorts.

Each cohort is biased by the strength of its students and the easiness of the grader. Assuming this is an additive effect, we back out of it the following way: we'll subtract the average score of the cohort on the first test, and add the average score of the cohort on the second test.

We compute an adjusted score $s'_1$ as follow

$$\forall j \leq p, \forall i \in A_j, s'_{1,i} = s_{1,i} - \frac{1}{|A_j|} \sum_{i \in A_j} ( s_{1,i} - s_{2,i} )$$

Finally, form a final score $s$ with whichever weighting you find appropriate

$$\forall i, s_i = \alpha s'_{1,i} + (1-\alpha) s_{2,i}$$

The downside is that an individual student might be penalized if the people in his cohort happened to get unlucky on the second test. But any statistical technique is going to carry this potentially unfair downside.

• As with any other proposal, this one will suffer from the inherent unfairness of being unable to distinguish the grader effect from the group effect. There simply is no way around that. At least your procedure is a little more transparent than some others that have been proposed, by making its arbitrary nature obvious (in the choice of $\alpha$). – whuber Nov 17 '14 at 16:47
• No - the cohorts aren't selected at random. – Scortchi - Reinstate Monica Nov 17 '14 at 17:11
• ... which, as @whuber keeps saying, is confounded with any inherent tendency of the cohort (owing to age or whatever) to do relatively better on one type of test than another. – Scortchi - Reinstate Monica Nov 17 '14 at 17:28
• You cannot eliminate confounding by taking larger cohorts! At best you can come up with ever more precise estimates of uninterpretable values. – whuber Nov 17 '14 at 17:31
• Reasonable, perhaps: but it's untestable given the information available to the OP. The validity of your answer relies on the truth of this implicit assumption. Even worse, its negation (which of course is also untestable) is eminently reasonable, too: because cohorts are self-selected, they may consist of people who perform in common ways on different assessment instruments, suggesting it may actually be likely that differential success will be due in part to the cohort and only partially due to variability among graders. – whuber Nov 17 '14 at 17:57

You can't. At least, not without collecting additional data. To see why, read @whuber's numerous upvoted comments throughout this thread.

### Edit

The problem solved in this answer is that of finding graders who give less points to the students they dislike.

### Original post

My approach, which I think is easy to implement, would be as follows:

Let $\mu_{k, i}$ denote the grade student $k$, belonging to cohort $i$ got on assignment 1. Let $y_{k, i}$ denote the grade for the second assignment.

## 1

Assume the model

$y_{k, i} = \mu_{k, i} + \alpha + \tau e_{k, i}$ and estimate $\alpha$ over all students. That is, $\alpha$ is estimated without regard to $i$. It is possible that $\alpha$ is zero but students may change their performance for the final exam giving an overall increase/decrease.

## 2

Let $G_i$ denote the generosity for the grader in cohort number $i$. Then form $\tilde{y}_{k, i}$ and assume the model

$y_{k, i} - \mu_{k, i} - \alpha = \tilde{y}_{k, i} = G_i + \sigma_i \tilde{e}_{k, i}$

And do 11 individual estimates of $G$ and $\sigma$

## 3

Now an unusual observation is one such that the quantity

$T = \vert \frac{\tilde{y} - G_i}{\sigma_i} \vert$ is large. Select the largest of these quantities for every cohort and investigate them.

### Note

All $e$'s are assumed to be Gaussian. The grades are not normally distributed so guidelines on the size of $T$ are difficult to give.

# R-code

Below is the code in R. Note that in your case, both mu and y will be given so the generating rows when they are assigned rnorm-numbers should be ignored. I include them to be able to evaluate the script without data.

mu_0 <- 50;
alpha <- 5;
tau<- 10;
# 0 Generate data for first assignment
mu <- matrix(rnorm(605, mu_0, tau), 11)

# 1 Generate data for second assignment and estimate alpha
G <- rnorm(11, 0)*10;
for(i in 1:11){
y[i,] <- rnorm(55, 0, sigma) + mu[i,] + alpha + G[i];
}

alpha_hat <- mean(y-mu)
alpha_hat

# 2 Form \tilde{y} and find unsual observations
ytilde <- y - mu - alpha_hat
T <- matrix(0, 11, 55);
for(i in 1:11){
G_hat <- mean(ytilde[i,]);
sigma_hat <- sd(ytilde[i,]);
T[i,] <- order(abs(ytilde[i,] - G_hat)/sigma_hat)
}
# 3 Look at grader number 2 by
T[2,]

• You do not seem to have answered the question: you end only with a recommendation to investigate any "unusual observations." How does that solve the problem? – whuber Nov 17 '14 at 0:18
• Reading the question again, perhaps I focused too much on the "individual" part. The problem solved in this answer is rather that of finding graders who give less points to the students they dislike. The original question is impossible(!) to solve. As already suggested, it is very likely that students collaborate or otherwise strongly correlate within each cohort. – Hunaphu Nov 17 '14 at 19:26

Rephrasing the problem: How best to approach setting a mark of a two part an exam with the conditions requiring that the second part is exposed to greater uncertainty due to the range of Delegated Markers' qualitative assessments.

Where: Master Tester = accountable person for exam Delegated Tester = person (1 of 11) assigned to mark par #2 of the exam Student = the guy that gets the fun of sitting an exam

Goals include: A) Students receive a mark that is reflecting their work B) Manage the uncertainty of the second part to align with the intent of the Master Tester

Suggested approach (answer): 1. Master Tester randomly selects a representative sample set of exams, marks the part #2 and develops correlation with the part #1 2. Utilise the correlation to assess all of the Delegated Markers' data (Part #1 vs #2 score) 3. Where the correlation is significantly different from the Master Tester - significance to be acceptable to the Master Tester - examine the exam as the Master Tester to re-assign the result.

This approach ensures that the Master Tester is accountable for the correlation and the acceptable significance. The correlation could be as simple as the score for part #1 vs #2 or relative scores for questions of test #1 vs #2.

The Master Tester will also be able to set a quality of result for Part #2 based on the "rubbery-ness" of the correlation.