Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
After reading How Not To Sort By Average Rating which deals with confidence interval for a Bernoulli parameter how would you extend it to more than two levels?
For example: Items are scored between 1 and 5 (1 is worst 5 is best). What is the best way to adjust the average score per item in order to take into account the number of scores it received (one 5 score should not give it an average of 5!)?
One way to cast your problem would be to treat it as a bayesian estimation problem.
Basically this means having a prior on your mean and update the mean based on each new observation over time.
A practical, yet theoretically disputable way to achieve this is to compute the mean as a function of the mean found in the corpus and the actual observations you have for this item. More precisely, in the recommender system setting, this could mean that you initialize the mean to the mean of the category of the item you're dealing with (in your example "statistics books" probably) and then update it each time a user gives a rating to this particular item.
You can design a clever update rule that has statistical foundations or rely on common sense to quickly produce a basic update rule like this one:
X : item
r_X^i : i-th rating for item X
C : all item in the same category as X, discarding empty ratings
mean_C = (1/|C|) * sum_{c in C} sum_{i} (r_c^i)
# when no rating => use category mean
mean_X^0 = mean_C
# when j ratings => ponderate category mean with actual ratings
mean_X^j = (1/n+1)(mean_C + sum_{i=1..n}(r_X^i))
When dealing in general with this kind of problems I recommend reading the work of Koren et al on the Netflix challenge. They grabbed quite a bit of performance by using unsupervised learning on user and content variables - the idea of using the category mean being a similar, yet naive cousin.
In the example you give, only one person has reviewed and given a score of 5/5. At this point, I would say you don't have enough information to give an informative estimate of the mean (or median). Possible scores are 1,2,3,4, or 5, so all you could say is that the mean average is somewhere between 1 and 5 and that one person on planet earth really likes the book.
However, if you have more people review, you can construct a confidence interval for that true mean review score. That way you could give a confidence level and some upper and lower bounds for the rating. (e.g. 95% confident that the book's rating is between 4.2 and 4.8). These bounds become tighter the more reviewers you have, so they do take into account the number of scores received.
However, typical Gaussian based confidence interval theory only holds up when you have a random sample from some population. Here the population is not well defined, perhaps those people who have bought the book through that website. Also, I would not say online reviewers are a random sample at all. I've found that book reviews (as with many online reviews) attract those people at the extremes who either love or hate the product. But perhaps it's best not to dwell too much on these issues...
I think what you're hinting at is the idea that if one person gave a book 5/5, this should probably not be considered better than an average of, say, 4.5/5 that's been reviewed by 200 people. And you mentioned "average", so perhaps you just want a one number summary that can be sorted easily.
I'm not too familiar with the Wilson score interval, but it looks like it is similar to the gaussian confidence interval, but it's construction is based on the score statistic.
You might want to look into some kind of weighted average that penalizes you for having a small sample.
