Is a product that has 4.9 stars from ten customers better than one that has 4.5 stars from a hundred customers?

Question

In many areas, we encounter a situation where we compare averages of highly skewed statistics using two unequally sized samples. Typically, this happens when comparing items in an online store. For example, as I outlined in the headline - one product has an excellent rating based on a small number of reviewers, the other has a slightly worse rating but still good, however it relies on a large number of reviews.

Are there any recommended or common practices to make comparisons based on this type of data?

Personally, I can think of sorting by lower bounds of confidence intervals or using Bayesian estimators. But I'm not sure if there isn't some common practice that is considered standard.

Answer 1

Clearly, several online platforms like Trip Advisor etc. have implemented a sorting system that trades these two things off. If they all did it the same way I'd guess one might consider that a standard approach. You can e.g. see that e.g. a restaurant with a rating based on more reviews is sometimes given as the best restaurant in town, even if a restaurant with a higher rating but very few reviews exists. Similarly, movie recommendations (sort of a rating/ranking for an average customer?). I think there have been data science competitions about these sort of things (worth looking up) and all sorts of publications (but not sure whether any big internet platform has disclosed exactly how they do it, might be an important competitive advantage - although perhaps for that reason they may have filed a patent that one could then search for?).

I can see many ways one could do this:

• Have a rating that you display + one that is calculated in the background where a certain number of average reviews (e.g. 2 or 3 out of 5) is added to calculate the average, i.e. with $N$ reviews with an average of $\bar{y}$ give a sorting average $\bar{y}' = (N \times \bar{y} + M \times y_\text{avg})/(N+M)$ for some $M>0$ and $y_\text{avg}$. That's basically a form of target encoding.
• Do a formal Bayesian analysis with priors (that could then even give you stuff like the probability that one "true" rating is in "truth" better than another etc.). E.g. if you ignore that the categories are ordered, you could work with a Dirichlet prior (e.g. a pretty uninformative Dirichlet(1/2, 1/2, 1/2, 1/2, 1/2, 1/2) for a 0 to 5 star scale) that allows very fast computations (also note: if these are not flexible enough, then mixtures of Dirichlet distributions can still be used with conjugate updating). For an ordinal model, I guess you'd have to use MCMC sampling (I might be wrong) so it would take longer to run. You could also consider the data as sort of continuous (they are not really, but it might give a useful enough result) or interval censored data (i.e. a rating of 5 really stands for a number in [4.5, 5]).
• Use a hierarchical model that shrinks the random effect of records with very few observations more towards the average than those with a lot of observations. That feels pretty logical to me. You can make this Bayesian and you can thereby avoid having to pick a prior that gets used for each item that is rated, but instead with enough items getting rated you get a data driven amount of shrinkage towards a data driven average (of course you still have to pick priors for the hyperparameters).
• Fit some kind of neural network with embeddings for items (could also have embeddings for raters, if one wants) and regularize the output so that for new unseen items an average rating comes out (and with enough reviews it converges to the average review, which is one way to solve the cold-start problem arising when there's not much data about an item or rater). That might be an overkill if you want to do only what you described, but is very handy for learning in a data driven way e.g. what movie to recommend to which customer (as it can - given enough data - create embeddings that capture more than one aspect about items). One good thing about neural networks is that you can have as your output and/or loss function just about anything you can think off (either a numeric rating that is constraint to (0, 5), or a probability for each separate categories, or a probability to have a rating at least as high as this category etc.), but of course in theory the same is the case for the other options (you can always write your own log-likelihood for any statistical model) although ordinal data is sufficiently common that most software can deal with it. Interpretability of the model parameters is of course more of an issue than with simpler models.

Note also that in practice I'd be quite worried about whether the ratings all measure the same underlying truth. Let's ignore that different raters like different things (but that's also a factor, how much should "It was the only restaurant that was open, but I really hate Italian food, so 0 stars." influence your opinion of a restaurant that offers Italian cuisine, if you really like Italian food?). Even if we ignore that, things can change over time and people might try to manipulate ratings. That could either be in the somewhat benign version where all your friends and family come to your newly opened restaurant and - of course - love it, or in the more malicious paid-for ratings by people that were never customers (but rate your restaurant highly - or even more maliciously just rate the local competition badly/write off-putting reviews about hygiene problems etc.).

That's perhaps only somewhat a statistical problem and needs other solutions (e.g. user registration instead of allowing anonymous ratings, processes to challenging ratings, surveillance for suspicious patterns in when reviews are posted and from where etc.). However, one can e.g. down-weight reviews by people that have not reviewed a lot or perhaps work with a hierarchical model (like for multiple-rater-multiple-case studies) that has a random effect for both rater and rated item - that could at least capture if some people tend to given on average higher or lower ratings than others (and you learn the most about item A compared with item B, if the same rater rates both of them).

You can further complicate such a model by introducing extra categories (ideally known a-priori like the cuisine(s) a restaurant says they offer - it would be much harder to infer such categories from the data unless there's a lot of data on all items and raters, although e.g. recommender systems try to do that) and then allowing for different raters to have different average rating tendencies for different categories. That could theoretically help with the issue of someone who just doesn't like Italian food rating an Italian restaurant, but in practice it may not help because you may not have many ratings from a rated. Even if they've rated many restaurants, they may not have rated enough Italian restaurants to show clearly that they don't like them because they presumably would self-select out of rating them by not eating in them (perhaps one could also model something regarding to what items people even buy/rate in the first place - not sure how though).

Answer 2

There are already excellent answers by Björn and jpa.

I will just give a basic philosophical thought that may help to put in perspective the many possibilities.

There is no unique truly best way of doing this. Different ways of doing it have different implications, and which of these are more or less desirable can neither be determined by the data nor by mathematical/statistical theory. It depends on what the user wants and how results are to be used. Rankings between rating distributions are constructed rather than "truly existing". They can be interpreted as "estimators of an existing unique true ranking" in a model framework, but more than one such framework is possible, and within such a framework the idea of an "unique true ranking" is still an implicit construction of the formal setup.

For example, as I outlined in the headline - one product has an excellent rating based on a small number of reviewers, the other has a slightly worse rating but still good, however it relies on a large number of reviews.

How to decide this may for example depend on what number of customers is required to make a product profitable. The question here is whether the "small number of reviewers" is so small that it makes you assume that the number of actual customers it points to is far from making the product profitable (involving also assumptions about how the number of reviewers relates to the number of customers). But then if we're talking ratings of films and we are film critics and not concerned about profit, this consideration may not be important. Another consideration is mean vs. variance and how difficult we want to make it for new products to achieve high ranks based on low reviewer numbers (as it often takes time until reviews come in). If we want a list where everything listed is rated very reliably, we need to demand a large number of reviewers to hold variance down. But then new products may not have a chance to be seen. If we want a list that gives interested people recommendations for trying out new good stuff, we may be happy with new products raising to the top early, and it'd be OK if many of them disappear from the top later when more raters rate it.

Answer 3

There are two separate probabilities involved:

1. A person that is shopping for this type of product ends up buying this particular product.
2. A person that bought this product leaves a positive review of it.

Assuming the products have been available for the same time, then the number of reviews (or purchases, if you track that) corresponds to 1., and the product quality corresponds to 2.

A product with good ratings but few purchases and reviews could be a product that is fit for very specific use-case, and the product description correctly indicates this.

What recommendation is best for a customer searching in this category?

Without further information, the probability of a good suggestion is P(1) ∩ P(2). For independent events, you can multiply the probabilities P(1) × P(2).

Assuming independence is a big assumption and can be distorted by e.g. affluent customers being more likely to buy expensive products and demand more of them. Often there is not enough data to make very detailed per-product estimates about such effects, but per-category some analysis could be performed.

If we count rating "5" as satisfied and anything less as unsatisfied, a rating average of 4.9 can be interpreted as P(2) = 90 %, and a rating average 4.5 as P(2) = 50 %. You could also count the number of 5-star reviews directly, but this way gives more weight to lower scores indicating "very dissatisfied".

From this simple model we would get 0.9 * 10 = 9 vs. 0.5 * 100 = 50, indicating that a random customer would benefit more from recommendation of the more popular product.

What recommendation is best for the shop?

The shop wants to keep their customers satisfied, but also wants to maximize the profit. Then the recommendation weight becomes P(1) × P(2) × Profit where the profit will vary between products.

Avoiding self-fulfilling bias

The most recommended products will get bought more and thus receive more reviews, leading to ever-increasing recommendation score.

To avoid this, you need to keep track of the number of times the product has been shown in search results and/or the number of times the product page has been opened.

It's a good idea to use the number of purchases for P(1) instead or in addition to the number of reviews.

Avoiding bias P(2) is not simple either, as not all satisfied customers leave a review. How likely they are to do so may vary by their wealth, amount of available time and the relevancy of any review perks offered. For positive reviews, a $1 discount on an expensive product is less effective than same perk for review of a cheap product. But if the customer is dissatisfied with an expensive product, they are much more likely to complain than for a cheap product.

Answer 4

I think the motivation for leaving a review matters most, success or failure in finding something that the reviewer thinks is hard to find. To that end, Amazon has added specific attributes for product reviews such as durability, ease of use, etc. That means there may be more than just two systems (ratios) to consider depending on the sub-ratings for the product attributes.

Accordingly, if I'm having a hard time finding an item that doesn't have a common design flaw (under powered, or plastic parts etc.), I may leave a positive review when I do find one, as a kind of public service.

Conversely if a product claims to have uniquely sturdy parts and I discover that it has the same design flaw as all the similar products, I'd warn the other consumers.

However, because the design flaw may not matter to all consumers, it could be a 5 star product to them, whereas for me, it's garbage.

An example would be a hook that says it can hold 50 lbs. but can only hold 30. If other consumers never use the hook to hold more that 2 lbs. to them, it's a great hook.

So the "answer" is really a matter of the product having been sold as a solution. All of the reviewers have to have the same problem such that they are evaluating the exact same solution.

As Björn pointed out in the restaurant example, just finding food wasn't a solution for that reviewer, the reviewer needed food that they liked.

Answer 5

There are already some decent answers, and I offer an additional perspective.

Contrary to what many people think, it is actually perfectly possible and not at all uncommon to use human judgement for quality control in manufacturing operations. Think: visual inspections to find scratches or dents or color defects, or tasting for food processing, or reporting the “feel” of fabrics.

At first glance, one might suspect this to be unreliable, poorly reproducible and extremely person-dependent. And that is correct, unless a lot of effort is put into training the inspectors and in controlling the environment.

Taste panels results need to be continuously monitored, to check consistency between inspectors. Visual identification of Scratches requires extreme control over lighting conditions, viewing distance, viewing time, and you need to check whether the inspector had the correct prescription glasses (if he needs any). Color needs to be judged in reference to color swabs, also at highly controlled lighting conditions and also controlled environment colors.

Now we return to the subject of online reviews, where NONE of these precautions are taken. In fact, there’s no guideline to be found on what “good quality” actually means, so you get things like the example in another answer: “I don’t like Italian food so a 0 for this Italian restaurant”.

You ask how to deal with small differences in unequal sample sizes. I posit that a full star difference on a five-point scale is mostly meaningless until you have at least several dozen reviews! At that point, different sample size becomes much less important. And even then, I advise against taking these review scores as absolute values.

It’s probably best to inspect the lowest scores, and read the motivations for these; that’s usually what I use in my decisions. Perhaps in these modern times with ChatGPT c.s., a summary on these motivations could be generated. I should patent that 😉