I have a data set of J raters who each give ratings to I objects (on say a scale of 1 to 5). My goal would be to construct some kind of overall rating for each i based on all of the raters' scores.
My first approach, which I have seen used before, is to construct a standardized score specific to each rater-object pair so that each of j's ratings are standardized and j's rating for object i is: (j's score on i - j's avg score on all I)/ (standard deviation of j's scores on all I). If all raters were equally trusted, I think this would be an acceptable approach and has a nice interpretation.
However, I suspect that there are lazy or poorly calibrated raters whose ratings are not spread out (e.g., they give 95% 4s and an occasional 5 when the distribution should really be uniform). It seems like I would want to somehow give greater weight to those raters with more spread out ratings. I also have an idea that a small subset of raters are "better" raters. Is there a way to perhaps weight each rater's score in line with their inter-rater reliability with the more accurate raters?
For simplicity, my preference would be to use some kind of generalized linear model, if possible.