I have a data set of J raters who each give ratings to I objects (on say a scale of 1 to 5). My goal would be to construct some kind of overall rating for each i based on all of the raters' scores.

My first approach, which I have seen used before, is to construct a standardized score specific to each rater-object pair so that each of j's ratings are standardized and j's rating for object i is: (j's score on i - j's avg score on all I)/ (standard deviation of j's scores on all I). If all raters were equally trusted, I think this would be an acceptable approach and has a nice interpretation.

However, I suspect that there are lazy or poorly calibrated raters whose ratings are not spread out (e.g., they give 95% 4s and an occasional 5 when the distribution should really be uniform). It seems like I would want to somehow give greater weight to those raters with more spread out ratings. I also have an idea that a small subset of raters are "better" raters. Is there a way to perhaps weight each rater's score in line with their inter-rater reliability with the more accurate raters?

For simplicity, my preference would be to use some kind of generalized linear model, if possible.

  • $\begingroup$ If you have the patience for it, generalizability theory will probably help you with this. $\endgroup$
    – rolando2
    Feb 1 '12 at 3:58
  • $\begingroup$ @rolando2, I probably don't have the patience for that and in this case need to go with something simple. Out of curiosity though, could you describe or provide a link that has a basic description of generalizability theory? $\endgroup$
    – d_a_c321
    Feb 1 '12 at 21:24
  • $\begingroup$ Loosely speaking, it's a form of variance partitioning. I don't think I ever read any single source on it that I really warmed up to. But the basic idea comes through in this example: you have scores assigned by various raters, on various items/questions, and in various conditions/modes/media. You want to know to what extent scores can be explained by, or are a function of, their rater, item, and condition. Generalizability theory is about a 40-year-old system for accomplishing this. $\endgroup$
    – rolando2
    Feb 2 '12 at 18:46

If the poorer raters are that bad, it suggests they are not adding information and could be dropped from the pool of raters. This would be preferable to weighting their ratings because:

  1. sometimes their "5"s will really be "5"s according to your better raters. Given that your better raters are providing all the information you need for an accurate rating, you don't need to incorporate the information from the poorer raters. Your results are not going to change for those objects.
  2. on objects that should have a lower rating than a "4" or a "5", you are obtaining information about what the rating "should" be from your better raters. To weight down the ratings from the poorer raters, you will base this on the differences between them and your good raters. Again, there appears to be no information gain from the poorer raters, as the final rating overall basically ignores their ratings.

Maybe I have missed something. However, if some of the ratings are basically useless, it is better to drop them entirely rather than try transformations - which aren't going to affect your overall ratings for each object anyways.

Update on comment: yes, exactly, the bad raters are "noise" that would need to be transformed to "signal". Given that any algorithm used to translate them to "signal" is based on the good raters and will only be approximate, there seems to be little point in going to this effort.

You could look at inter-rater reliability measures for the better poor raters and see what transpires. There are a number of factors to take into account even with this reduced approach:

  1. If there are a lot of items that are rated at the extremes by your good raters ("1"s and "5"s) and your other raters are managing to give equivalent extreme ratings, the inter-rating reliability measure will be affected by these extreme value objects, and actual inter-rater reliability may be lower.
  2. You could still get poor inter-rater reliability measures even with the subset who are "less bad".

So this is a path you could go down, and be prepared that you may not get a good result even with your subset.

To reframe this, removing the bad rater scores is not throwing away data, it is throwing away noise.

  • $\begingroup$ Thanks for the answer. For 1) you mean to say that the true answer won't change... right? If I mix and incorporate the bad raters, I'd just get noise. For 2), it seems like again there is the chicken and egg problem. If I calibrate the bad reviewers sufficiently using the good reviewers, this also doesn't add information.... The one thing I wanted to try was to see if say the top quartile of crappy raters have a high kappa (or correlation) with the really trusted ones. If so, could I somehow use that subset of "bad" raters? Any further thoughts would be appreciated. $\endgroup$
    – d_a_c321
    Feb 1 '12 at 21:20

If I am understanding correctly, you can analyze your data with a simple random intercept model. You have raters indexed by j from 1 to J and items indexed by i from 1 to I. For each item, each rater produces a response $ R_{ij} $ Using the terminology of psychometrics, it seems that you want to estimate the "difficulty" (or quality) of each item. You can estimate it using the following model:

$$ (R_{ij} = x | \zeta_j) = \zeta_j + \delta_i + \epsilon_{ij} \\ $$

$$ \zeta_j \sim N(0,\psi) \\ \epsilon_{ij} \sim (0,\theta) $$

Using this model, the interpretation is as follows:

$ \delta_i $ stands are fixed effects that will represent the difficulty/quality associated with each one the items.

$ \zeta_j $ will by the (random) rater intercept. Depending on what software do you use to estimate the model you may or may not get this automatically as part of the output.

$ \psi $ will be the variance of the random effect (the variance of your raters).

$ \theta $ wil be the residual variance.

If you are interested in the reliability of your raters, you can calculate the intra class correlation based on the two variance parameters:

$$ ICC = \dfrac{\psi}{\theta+\psi} $$

This kind of model should be easily estimable in any statistical package. For instance, in Stata you can use xtreg and in R you can use the llme4 package.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.