I have a collection of scores from critics for various restaurants. Some restaurants are rated by multiple critics and some by only a single critic. Each critic rates restaurants using different scales such as 1 - 5 (stars), 1 - 10, 1 - 100 etc. Most critics rate a restaurant only once although they may revisit it at a certain point (probably this complication can be ignored and I will use just the most recent rating).
Also some critics only use a subset of the range (ie 60 - 100). This may be due to the fact that they only review reasonably good restaurants and hence don't use the bottom end of the range, or possibly other reasons.
In addition the distribution of ratings given by an individual critic are often quite skewed (left or right).
Can anyone suggest some good ways of combining individual critics scores into a single score ?
I thought about standardizing each critics score and then taking the average. However this creates a couple of problems:
There is the case where multiple critics give a high rating and one might give a very low rating. If so should I ignore the outlier or give them equal weighting? Or should I assign a given critic some sort of reliability rating and use that to create a weighted average. How might one calculate this reliability ?
I'm not sure I should drop the information that a critic only uses a subrange of the scale such as 60 - 100 instead of 0 - 100 as this may reflect the fact that they are being selective about which restaurants they review. In other words an above average z-score for critic A (who picks only good restaurants), may mean something quite different than an above average z-score for critic B (who dines at a wider range of restaurants).
Sorry for the vague question, but does anyone have any suggestions about how I approach this ?
Is there anyway to use the fact that I have multiple ratings for a large number of restaurants to somehow calibrate the critics' ratings to a common scale ? Or is this nonsensical.
Should I be looking at some sort of random effects model ?
Thanks for any pointers or suggestions.