this is just a little curiosity of mine, but has anyone heard of some simple model like Laplace's law of succession, however for ordinal RVs instead of a binomial? Specifically, I'm thinking about this in the context of e.g. product reviews online: is there some nice simple rule of thumb for determining whether it's likely that e.g. product A with an average rating of 4.5 stars based on 10 reviews really is better than product B with 4.3 stars based on 100 reviews?

I know that Laplace's law of succession works because the Beta-Binomial model is especially nice and simple. In the ratings problem, we have two ordinal RVs with 5 levels each so that's a bit more complicated. If we had the number of ratings per each number of stars (e.g. 10 1-star reviews, 25 2-star, ...) we could simplify things by ignoring the rounding and doing a t-test (or a Bayesian equivalent of that), or even set up some more complicated but still Normal model which accounts for the rounding. But I'm trying to think in the most basic case, where all we have is the average rating $\bar x_i$ and the number of reviews $n_i$, is there anything we can do to give us some kind of rule of thumb (while perhaps making some strong assumptions)?

  • $\begingroup$ If you assume that all ratings are either 4 or 5 stars, and replace “really is better” with “has reviews which were sampled from a population with more frequent 5-star ratings”, then the problem is tractable. $\endgroup$
    – Matt F.
    Commented Jul 7, 2023 at 14:44
  • $\begingroup$ The mean and count are not sufficient to answer the question, because any (reasonable) answer must also depend on the distribution of the ratings. For instance, a mean of 4.5 with ten reviews could be nine fives and one zero or it could be five fives and five fours: surely those convey different information to the user, with the latter suggesting greater consistency than the former. Sure, you could make some "strong assumptions," but then what you get out of the exercise might be no more than what you put into it through those assumptions. $\endgroup$
    – whuber
    Commented Jul 7, 2023 at 16:04


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.