Combining ratings from multiple raters of different accuracy

Question

I have a data set of J raters who each give ratings to I objects (on say a scale of 1 to 5). My goal would be to construct some kind of overall rating for each i based on all of the raters' scores.

My first approach, which I have seen used before, is to construct a standardized score specific to each rater-object pair so that each of j's ratings are standardized and j's rating for object i is: (j's score on i - j's avg score on all I)/ (standard deviation of j's scores on all I). If all raters were equally trusted, I think this would be an acceptable approach and has a nice interpretation.

However, I suspect that there are lazy or poorly calibrated raters whose ratings are not spread out (e.g., they give 95% 4s and an occasional 5 when the distribution should really be uniform). It seems like I would want to somehow give greater weight to those raters with more spread out ratings. I also have an idea that a small subset of raters are "better" raters. Is there a way to perhaps weight each rater's score in line with their inter-rater reliability with the more accurate raters?

For simplicity, my preference would be to use some kind of generalized linear model, if possible.

Answer 1

If the poorer raters are that bad, it suggests they are not adding information and could be dropped from the pool of raters. This would be preferable to weighting their ratings because:

1. sometimes their "5"s will really be "5"s according to your better raters. Given that your better raters are providing all the information you need for an accurate rating, you don't need to incorporate the information from the poorer raters. Your results are not going to change for those objects.
2. on objects that should have a lower rating than a "4" or a "5", you are obtaining information about what the rating "should" be from your better raters. To weight down the ratings from the poorer raters, you will base this on the differences between them and your good raters. Again, there appears to be no information gain from the poorer raters, as the final rating overall basically ignores their ratings.

Maybe I have missed something. However, if some of the ratings are basically useless, it is better to drop them entirely rather than try transformations - which aren't going to affect your overall ratings for each object anyways.

Update on comment: yes, exactly, the bad raters are "noise" that would need to be transformed to "signal". Given that any algorithm used to translate them to "signal" is based on the good raters and will only be approximate, there seems to be little point in going to this effort.

You could look at inter-rater reliability measures for the better poor raters and see what transpires. There are a number of factors to take into account even with this reduced approach:

1. If there are a lot of items that are rated at the extremes by your good raters ("1"s and "5"s) and your other raters are managing to give equivalent extreme ratings, the inter-rating reliability measure will be affected by these extreme value objects, and actual inter-rater reliability may be lower.
2. You could still get poor inter-rater reliability measures even with the subset who are "less bad".

So this is a path you could go down, and be prepared that you may not get a good result even with your subset.

To reframe this, removing the bad rater scores is not throwing away data, it is throwing away noise.

Answer 2

If I am understanding correctly, you can analyze your data with a simple random intercept model. You have raters indexed by j from 1 to J and items indexed by i from 1 to I. For each item, each rater produces a response $ R_{ij} $ Using the terminology of psychometrics, it seems that you want to estimate the "difficulty" (or quality) of each item. You can estimate it using the following model:

$$ (R_{ij} = x | \zeta_j) = \zeta_j + \delta_i + \epsilon_{ij} \\ $$

$$ \zeta_j \sim N(0,\psi) \\ \epsilon_{ij} \sim (0,\theta) $$

Using this model, the interpretation is as follows:

$ \delta_i $ stands are fixed effects that will represent the difficulty/quality associated with each one the items.

$ \zeta_j $ will by the (random) rater intercept. Depending on what software do you use to estimate the model you may or may not get this automatically as part of the output.

$ \psi $ will be the variance of the random effect (the variance of your raters).

$ \theta $ wil be the residual variance.

If you are interested in the reliability of your raters, you can calculate the intra class correlation based on the two variance parameters:

$$ ICC = \dfrac{\psi}{\theta+\psi} $$

This kind of model should be easily estimable in any statistical package. For instance, in Stata you can use xtreg and in R you can use the llme4 package.