# Aggregate opinions to a single measure

I have a following situation in which 6 experts are asked to give opinion about a bottle of wine by rating it from 1 - 5 stars. In addition, somehow we have a measure of the informativeness of each rating (think of it as a measure of how reliable the expert who gives the rating is), and let's call this measure the informativeness score of an expert

We want to aggregate these 6 opinions into a single measure of the quality of the wine, so what is the most sensible method of doing it?

My initial thought was to normalize the informativeness scores among the experts to sum up to 1, and just sum the products between the informativeness score and the corresponding rating over all the experts. However, this approach suffers from the fact that a low-score high-value rating may overwhelm a high-score low-value rating during the aggregation.

-
Hi. Could you please describe what you mean by the reliability of the expert. Is a reliable expert one who (1) consistently provides similar ratings across wines of similar quality or (2) is more experienced and you believe their ratings are more accurate? Or do you mean something else? – Michelle Jan 23 '12 at 22:04
@Michelle, it's (2) mostly: if an expert is regarded more reliable (i.e. with a high informativeness score), then he/she is regarded as more experienced and skillful in judging the wine quality, hence I believe his/her rating would be more accurate. Thanks. – MLister Jan 23 '12 at 22:08
Your 'initial thought' to form a weighted mean sounds sensible to me. I don't understand your misgiving. – onestop Jan 23 '12 at 22:14
I'm wondering if the issue is that your informativeness score is not as reflective of differences between raters as you would like. For example, if you look at your raw informativeness scores, is a rater who is "half as good" as another rater have an informativeness score that is half as large? – Michelle Jan 23 '12 at 22:16
If you take the weighted mean of a rating of 5 with a weight of 0.05 with a rating of 1 and a weight of 0.25, you get (5*0.05 + 1* 0.25)/(0.05+0.25) = 1.67. That's a lot nearer 1 than 5, so the low-score high-value rating hasn't overwhelmed the high-score low-value rating at all. – onestop Jan 24 '12 at 12:52

A weighted mean, using the informativeness (horrible word!) as weights is a sensible approach.

You don't need to worry about the low-score high-value rating appearing to be equivalent to a high-score low-value rating. The weighted mean is the sum of weights x score, divided by the sum of the weights. Although the low-score high-value rating and high-score low-value rating might both make the same contribution to the sum in the numerator, the high-value rating adds more to the sum in the denominator and hence it actually makes more of a contribution.

-

This question & answer discussion has a peculiar character to it. Both @PeterEllis and @Michelle have provided decent answers (your question was also addressed by @onestop in the comments). Your question and your responses to these answers indicate that you need to think through what you are asking more thoroughly.

For example, when you say that "a low-score high-value rating (e.g. a rating of 5 with an informativeness score 0.05/1) might be equivalent or even higher than a high-score low-value rating (e.g. a rating of 1 with an informativeness score 0.25 or a rating of 1 with an informativeness score 0.2/1)", you evidence a lack of understanding. (I mean no disrespect here.) If those 'informativeness scores' are accurate, this is exactly what is supposed to happen and it will optimally incorporate the information in the ratings and the information in the scores according to that scheme. You can question the accuracy of the 'informativeness scores' or prefer a different scheme, but once you've ascented to these, that is the answer.

@Michelle offers a potential alternative scheme, that may be more appropriate depending on what you want to optimize. It is odd to criticize this by saying that "the rating picked by the high-skill expert is not taken into account in any way in the final result", because that is the point of this scheme.

This all reminds me of a famous (cranky) paper by Guttman, What is Not What in Statistics:

48. Permission is not required in data analysis

What is required is a loss function to be minimized. Practitioners like to ask about a priori rules as to what is "permitted" to be done with their unordered, ordered, or numerical observations, without reference to any overall loss function for their problem. Instead, they should say to the mathematician: "Here is my loss function: how do I go about minimizing it ?"

It sounds to me like you need to figure out what you think is the optimal way to incorporate multiple sources of information with these characteristics. 'Optimal' depends on your goals, and the other stuff falls out from that. The Sage monograph Summated Rating Scale Construction: An Introduction may help you to think through these issues.

On the other hand, if your real concern is the validity of the 'informativeness score', this website might help you think through the pros and cons of different approaches to inter-rater agreement.

I apologize if it seems like this has a hectoring tone; I don't mean to come across that way. But your questions cannot be answered until you clarify these issues for yourself.

-

Based on the information that you provide, I think you're after a consensus-type measure, where the number of experts agreeing at each star level is weighted by how much you believe their ratings are accurate.

So, for example, say your 6 raters have the following "expert" weights:

• Raters A and B each have a weight of 1
• Rater C has a weight of 0.8
• Rater D has a weight of 0.5
• Raters E and F each have a weight of 0.25

Now consider one bottle of wine. You receive the following ratings:

Rater     Rating
A         5
B         4
C         4
D         5
E         4
F         4


Rating 5 is based basically on 1.5 "reliable" Raters (1(A) + 0.5(D)) Rating 4 is based basically on 2.3 "reliable" Raters (1(B)+0.8(C)+0.25(E)+0.25(F)) so the consensus is 4 stars.

We don't adjust the rating based on rater reliability, we adjust the number of "experts" giving each rating based on rater reliability. Then we pick the rating based on the most number of experts.

Update based on comments and the answer from @gung: if the experts are varying wildly across wines, and therefore you get an aggregate score that you don't think is a good summary of the wine, the problem is likely to lie with your expert panel rather than the method used for aggregation. If the expert ratings are not reliable or are invalid for reasons other than unreliability, it won't matter what aggregation method you use because the GIGO principle will apply.

-
 thanks for your answer. If I understand correctly, you sum the expert informativeness scores behind each given rating, and pick out the one that has the largest sum, i.e. the one that gathers the highest level of informativeness. – MLister Jan 24 '12 at 15:05 It is an interesting approach, but a potential problem is that if you have 5 low-skill experts who endorse the same rating versus a high-skill expert who rate differently, then it is possible that the sum of the informativeness scores of the 5 low-skill experts may exceed the high-skill one, and the rating picked by the high-skill expert is not taken into account in any way in the final result. – MLister Jan 24 '12 at 15:06 Where you see a problem, I see a strength. :) I'm assuming you do have experts, and not people off the street. Therefore, the quality of judgement between them is not as great as between members of the general population. If 5 lower skill experts (who are still experts) give you one specific rating, and 1 higher skill expert gives you a completely different one, I would wonder about the rating from the higher skill expert. This approach does assume that your experts are, in fact, expert. – Michelle Jan 24 '12 at 21:15