In a class with multiple teachers, how can I transform student scores based on their teacher's average compared to the population average?

Question

It has been ages since I've taken any statistics courses, and I have found myself in the following situation:

I am in charge of a university course with about 400 students and 10 assessors.

There is a presentation-style activity coming up, during which any given student will be graded by one of the assessors. Each assessor will have about 30, 40 students.

We have a rubric for grading, but historically there is a lot of variation in each assessor's average, standard deviation, range, etc. Teaching beliefs and expectations of students are very hard to change!

I would like to "normalize" student scores, so that any given student's assessor's mean would be compared to the population mean, and then that student's score would be scaled up or down accordingly.

Ideally, perfect scores (say, 20/20) would not change. I imagine as scores get closer to full points, the shift up or down would be less significant. Lower scores will be scaled a lot more.

To me, this sounds logarithmic, but I don't have the slightest idea how to go about it.

I am making a spreadsheet to handle this, so I was hoping for some mathematical formula to use. I am quite adept with Excel, and I have a strong mathematical background (at least I thought I did!), so complexity of the formula is not really a problem.

I don't remember a whole lot of terminology, however, so please be patient with me! And please let me know if more details are needed.

Cheers!

Answer 1

If you assume each assessors' scores are normally distributed, then you can compute standardized scores for each student as

\begin{equation} z=\frac{x-\mu}{\sigma} \end{equation}

where x is the student's score, $\mu$ is the mean score of the student's assessor, and $\sigma$ is the standard deviation of the student's assessor's scores. Here is an example in R.

# Simulate 40 generous and 40 grumpy assessor scores
set.seed(23)
x1 <- rnorm(n=40, mean=13, sd=3)
x2 <- rnorm(n=40, mean=8, sd=2)

# Standardize student scores based on their assessor
z1 <- (x1 - mean(x1))/sd(x1)
z2 <- (x2 - mean(x2))/sd(x2)

# Plot distributions of unstandardized and standardized scores
layout(matrix(1:4, 2, 2))
hist(x1, main="Generous assessor- unstandardized", xlim=c(4, 22))
abline(v=mean(x1), col="red")
hist(z1, main="Generous assessor- standardized", xlim=c(-3, 3))
abline(v=mean(z1), col="red")
hist(x2, main="Grumpy assessor- unstandardized", xlim=c(4, 22))
abline(v=mean(x2), col="red")
hist(z2, main="Grumpy assessor- standardized", xlim=c(-3, 3))
abline(v=mean(z2), col="red")

You can see that the unstandardized scores are biased by the assessor, but the distributions of standardized scores are roughly normally distributed around 0.