Aggregate opinions to a single measure

Question

I have a following situation in which 6 experts are asked to give opinion about a bottle of wine by rating it from 1 - 5 stars. In addition, somehow we have a measure of the informativeness of each rating (think of it as a measure of how reliable the expert who gives the rating is), and let's call this measure the informativeness score of an expert

We want to aggregate these 6 opinions into a single measure of the quality of the wine, so what is the most sensible method of doing it?

My initial thought was to normalize the informativeness scores among the experts to sum up to 1, and just sum the products between the informativeness score and the corresponding rating over all the experts. However, this approach suffers from the fact that a low-score high-value rating may overwhelm a high-score low-value rating during the aggregation.

Answer 1

A weighted mean, using the informativeness (horrible word!) as weights is a sensible approach.

You don't need to worry about the low-score high-value rating appearing to be equivalent to a high-score low-value rating. The weighted mean is the sum of weights x score, divided by the sum of the weights. Although the low-score high-value rating and high-score low-value rating might both make the same contribution to the sum in the numerator, the high-value rating adds more to the sum in the denominator and hence it actually makes more of a contribution.

Answer 2

This question & answer discussion has a peculiar character to it. Both @PeterEllis and @Michelle have provided decent answers (your question was also addressed by @onestop in the comments). Your question and your responses to these answers indicate that you need to think through what you are asking more thoroughly.

For example, when you say that "a low-score high-value rating (e.g. a rating of 5 with an informativeness score 0.05/1) might be equivalent or even higher than a high-score low-value rating (e.g. a rating of 1 with an informativeness score 0.25 or a rating of 1 with an informativeness score 0.2/1)", you evidence a lack of understanding. (I mean no disrespect here.) If those 'informativeness scores' are accurate, this is exactly what is supposed to happen and it will optimally incorporate the information in the ratings and the information in the scores according to that scheme. You can question the accuracy of the 'informativeness scores' or prefer a different scheme, but once you've ascented to these, that is the answer.

@Michelle offers a potential alternative scheme, that may be more appropriate depending on what you want to optimize. It is odd to criticize this by saying that "the rating picked by the high-skill expert is not taken into account in any way in the final result", because that is the point of this scheme.

This all reminds me of a famous (cranky) paper by Guttman, What is Not What in Statistics:

48. Permission is not required in data analysis

What is required is a loss function to be minimized. Practitioners like to ask about a priori rules as to what is "permitted" to be done with their unordered, ordered, or numerical observations, without reference to any overall loss function for their problem. Instead, they should say to the mathematician: "Here is my loss function: how do I go about minimizing it ?"

It sounds to me like you need to figure out what you think is the optimal way to incorporate multiple sources of information with these characteristics. 'Optimal' depends on your goals, and the other stuff falls out from that. The Sage monograph Summated Rating Scale Construction: An Introduction may help you to think through these issues.

On the other hand, if your real concern is the validity of the 'informativeness score', this website might help you think through the pros and cons of different approaches to inter-rater agreement.

I apologize if it seems like this has a hectoring tone; I don't mean to come across that way. But your questions cannot be answered until you clarify these issues for yourself.

Answer 3

Based on the information that you provide, I think you're after a consensus-type measure, where the number of experts agreeing at each star level is weighted by how much you believe their ratings are accurate.

So, for example, say your 6 raters have the following "expert" weights:

• Raters A and B each have a weight of 1
• Rater C has a weight of 0.8
• Rater D has a weight of 0.5
• Raters E and F each have a weight of 0.25

Now consider one bottle of wine. You receive the following ratings:

Rater     Rating
A         5
B         4
C         4
D         5
E         4
F         4

Rating 5 is based basically on 1.5 "reliable" Raters (1(A) + 0.5(D)) Rating 4 is based basically on 2.3 "reliable" Raters (1(B)+0.8(C)+0.25(E)+0.25(F)) so the consensus is 4 stars.

We don't adjust the rating based on rater reliability, we adjust the number of "experts" giving each rating based on rater reliability. Then we pick the rating based on the most number of experts.

Update based on comments and the answer from @gung: if the experts are varying wildly across wines, and therefore you get an aggregate score that you don't think is a good summary of the wine, the problem is likely to lie with your expert panel rather than the method used for aggregation. If the expert ratings are not reliable or are invalid for reasons other than unreliability, it won't matter what aggregation method you use because the GIGO principle will apply.