Simple up/down vote rating but weighted by number of responses

Question

I am trying to analyse the ratings for restaurants from a website. The rating system on the website is pretty simple: people can up-vote or down-vote.

The restaurant is then presented to website users with the number of votes it received and a percentage score (I am assuming this is simply an average of the votes: e.g. 2 up-votes and 1 down-vote will give a score of 66.7%).

Intuitively if I were to pick a restaurant to visit: I would rather pick one with a score of 90% and 100 votes than pick one with a score of 100% and 1 vote. Even though the former score is lower, I also know that the score won't be drastically affected by the next vote so I trust the score more. (Something Bayesian about this?)

How can a third metric (rating) be derived that combines score and number of votes to order restaurants not only by score but also by how trustworthy that score is?

I can think of a few empirical ways of doing this but they often result in me asking myself: What's better between (75%, 10 votes) and (80%, 8 votes)? Which isn't as obvious as the example above. Is there a more formal way to answer this question?

Answer 1

If I understand your question correctly, you would like to combine both the point estimate (percentage of up-votes) and its reliability (roughly represented by the total number of votes) in a single number.

As confidence intervals represent both values, a very simple approach would be to base the score on the lower limit of a (e.g. 50%) confidence interval for a binomial proportion $\hat{p}$. To make sure that this value is positive, you should use a confidence interval that also works with small values of $n$ and also with values of $p$ close to one ore zero, e.g. the Bayesian HPD interval, the "exact" Clopper-Pearson interval, or the WIlson interval. The latter has the attractive feature of being computable in closed form, i.e., the lower limit is: $$p_{lower} = \frac{1}{1+z^2/n}\left[\hat{p} + \frac{z^2}{2n} - z \sqrt{\frac{\hat{p}(1-\hat{p})}{n}+\frac{z^2}{4n^2}}\right] $$ where $z=z_{1-\alpha/2}$ is the $(1-\alpha/2)$ standard normal quantile; for $1-\alpha=50\%$, it is $z\approx 0.7$.

Answer 2

(Something Bayesian about this?)

No.

A large number of experiments, eliminates randomness, which is the measure of uncertainty of an outcome. So, essentially randomness at play is reduced and the rating of the restaurant becomes more reliable with more votes.

How can a third metric (rating) be derived that combines score and number of votes to order restaurants not only by score but also by how trustworthy that score is?

The metric will largely depend on the objective for deriving the metric.

One metric could be -- that takes into account the number of votes -- could be $(n_+/N)\ln{N}$ where $n_+$ is the number of positive votes, $\ln{}$ is natural log, $N$ is the total number of votes, $N \geqslant1$.