I'm working on a personal project to rank user ratings on a 5 star scale and have implemented Evan Miller's often-referenced Wilson Score Interval calculation with the help of this comment. My data just contains the rating distribution for each item, so I can't segment by any particular user for some advanced approaches I've seen mentioned.
Something I'm noticing is that many of the individual items have overlapping confidence intervals. When deciding on what the "best" (or worst) items are,
- Is the lower Wilson bound enough?
- Does it matter that the CIs overlap?
- If the CIs are not distinct, is/should there be a tie breaker?
- What tie breakers should be used?
PS: As a side note on various approaches for ranking user ratings, I picked the Wilson approach because the derived value is only dependent on the ratings collected for the individual item. No need to find averages across the whole dataset or or come up with some hastily chosen multiplier like you see with IMDB's calculation using Bayesian. My data comes from forum discussion posts going back 10+ years, and the number of ratings have fluctuated over time, so taking averages/medians across that whole time leads to unrealistic results.