6
$\begingroup$

After reading How Not To Sort By Average Rating which deals with confidence interval for a Bernoulli parameter how would you extend it to more than two levels?

For example: Items are scored between 1 and 5 (1 is worst 5 is best). What is the best way to adjust the average score per item in order to take into account the number of scores it received (one 5 score should not give it an average of 5!)?

Shame on you Amazon!

$\endgroup$
  • $\begingroup$ If you are using the average rating as a dependent variable, this sounds like a heteroskedasticity issue, since averages based on smaller samples sizes will be more variable. For example, if you're using regression you can weight observations (e.g. en.wikipedia.org/wiki/…) to take care of this. $\endgroup$ – Macro Apr 13 '12 at 3:26
  • 1
    $\begingroup$ see this related question: stats.stackexchange.com/questions/1848/… $\endgroup$ – Jeromy Anglim Apr 13 '12 at 6:02
1
$\begingroup$

One way to cast your problem would be to treat it as a bayesian estimation problem.

Basically this means having a prior on your mean and update the mean based on each new observation over time.

A practical, yet theoretically disputable way to achieve this is to compute the mean as a function of the mean found in the corpus and the actual observations you have for this item. More precisely, in the recommender system setting, this could mean that you initialize the mean to the mean of the category of the item you're dealing with (in your example "statistics books" probably) and then update it each time a user gives a rating to this particular item.

You can design a clever update rule that has statistical foundations or rely on common sense to quickly produce a basic update rule like this one:

X : item 
r_X^i : i-th rating for item X  
C : all item in the same category as X, discarding empty ratings
mean_C = (1/|C|) * sum_{c in C} sum_{i} (r_c^i)
# when no rating => use category mean
mean_X^0 = mean_C 
# when j ratings => ponderate category mean with actual ratings
mean_X^j = (1/n+1)(mean_C + sum_{i=1..n}(r_X^i)) 

When dealing in general with this kind of problems I recommend reading the work of Koren et al on the Netflix challenge. They grabbed quite a bit of performance by using unsupervised learning on user and content variables - the idea of using the category mean being a similar, yet naive cousin.

$\endgroup$
  • 1
    $\begingroup$ The entire point of that article is that an estimate of a mean, no matter how it is arrived at, is only part of the picture. Uncertain estimates perhaps should not be given as much "weight" as more certain ones. The Bayesian approach really does not address this issue. $\endgroup$ – whuber Apr 13 '12 at 15:11
  • 1
    $\begingroup$ Right, more certain estimates should be given more weight. I suspect there are means to render this notion in the bayesian framework by playing with the risk functions. From an application perspective, you can naively incorporate such a notion by discarding (or down-weighting) items with less than a pre-defined number of reviews. You can also incorporate other elements like date of reviews, reviewers profiles etc. At a certain point you might want to switch to a Learning to Rank approach to combine all those elements. $\endgroup$ – oDDsKooL Apr 15 '12 at 8:43
  • 1
    $\begingroup$ Your reference to that Wikipedia article addresses the fundamental flaw in the "How not to Sort by Average Rating" article alluded to in the question. That article proposes a procedure that seems to have statistical merit because it incorporates confidence intervals. However, its approach--although it might accidentally work ok in some circumstances--is arbitrary. $\endgroup$ – whuber Apr 15 '12 at 14:03
0
$\begingroup$

In the example you give, only one person has reviewed and given a score of 5/5. At this point, I would say you don't have enough information to give an informative estimate of the mean (or median). Possible scores are 1,2,3,4, or 5, so all you could say is that the mean average is somewhere between 1 and 5 and that one person on planet earth really likes the book.

However, if you have more people review, you can construct a confidence interval for that true mean review score. That way you could give a confidence level and some upper and lower bounds for the rating. (e.g. 95% confident that the book's rating is between 4.2 and 4.8). These bounds become tighter the more reviewers you have, so they do take into account the number of scores received.

However, typical Gaussian based confidence interval theory only holds up when you have a random sample from some population. Here the population is not well defined, perhaps those people who have bought the book through that website. Also, I would not say online reviewers are a random sample at all. I've found that book reviews (as with many online reviews) attract those people at the extremes who either love or hate the product. But perhaps it's best not to dwell too much on these issues...

I think what you're hinting at is the idea that if one person gave a book 5/5, this should probably not be considered better than an average of, say, 4.5/5 that's been reviewed by 200 people. And you mentioned "average", so perhaps you just want a one number summary that can be sorted easily.

I'm not too familiar with the Wilson score interval, but it looks like it is similar to the gaussian confidence interval, but it's construction is based on the score statistic.

You might want to look into some kind of weighted average that penalizes you for having a small sample.

$\endgroup$
  • $\begingroup$ I originally wanted to find a quantity to rank items for recommendation purposes and an average seemed a reasonable idea. I am not sure how I would use that adjusted average for other purposes outside sorting. $\endgroup$ – Ηλίας Apr 13 '12 at 16:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.