What is an appropriate test to see if there is a relationship between two things, where one of them is itself obtained by averaging human rankings.

As an example, there are good and bad bottles of wine. The company scientist has found a possible objective measure of the quality of the wine. To validate this measure, gather 100 different bottles, and ask 10 people to rate each bottle on a 1-5 scale. Then average the ratiings, so each bottle has a single average score A. It also has an objective score B given by the scientist's measure, which is on a continuous scale.

One could do correlation between A,B across the 100 bottles, or alternately gather some of the highly human-rated wines in group A1, some low-rated wines in group A2, and then do a t-test of difference of means of the scientist measure on groups A1 vs A2.

But neither of these take into account the fact that the ratings A were themselves obtained by averaging, which has its own variance.

(To explain the question further, suppose the wine bottles were rated on a 1-1000 scale rather than a 1-5 scale. Consider two bottles, one has ratings of between 498 and 502 with an average rating of 500, and the second has an average rating of 520 with similar small variance. The objective measure also gives the second bottle a higher score, so this example is weak support for a relationship. But now suppose that the ratings of the first bottle ranged from 1 to 1000, with an average of 500, and the ratings of the second also had huge variance. In this case the difference in means seems accidental, and this pair of (A,B) should provide less support for the proposed relationship)

How to account for this?

  • 1
    $\begingroup$ Is not using the average an option? Is it permissible to use the raw data? $\endgroup$ – Sal Mangiafico Jan 13 '19 at 17:02
  • 1
    $\begingroup$ This is an FAQ that has received no good answers. It's the same question as how to rank ratings given to products on Web sites, for instance. The main reason it has no good answers is that any answer ultimately will reflect a trade-off between the sizes of the differences and the uncertainties in them. That trade-off reflects the degree to which you might be risk-adverse or risk-seeking and therefore introduces a personal, subjective element into the problem. $\endgroup$ – whuber Jan 13 '19 at 19:33
  • $\begingroup$ Some solutions based on multi-criterion decision analysis are discussed at stats.stackexchange.com/questions/9137 and stats.stackexchange.com/questions/3201 $\endgroup$ – whuber Jan 13 '19 at 19:36
  • $\begingroup$ @Sal Mangiafico Yes using the individual raw rankings rather than the average is possible. However, we don't know how to do this. Simply doing correlation of the individual rankings against the scientist's measure would tell us how each person correlates with the objective measure. That would be useful if the objective measure was proven and we wanted to identify a couple human raters. $\endgroup$ – largewords Jan 13 '19 at 23:54
  • $\begingroup$ But in our case, we're not interested in the individual people, only in the using the overall human-rated quality of a bottle of wine to validate the objective measure, where each bottle is ranked by a small group of people (we do not care about their individual opinions). $\endgroup$ – largewords Jan 13 '19 at 23:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.