12
$\begingroup$

I need an idea for a new rating system.. the problem in the ordinary one (just average votes) is that it does not count how many votes...

For example consider these 2 cases:

3 people voted 5/5

500 people voted 4/5

The ordinary voting systems just take the average, leading to the first one to be better.. However, I want the second one to get a higher rating, because many people have voted for the second...

Any help?

Improve this question
$\endgroup$
2

6 Answers 6

Reset to default
13
$\begingroup$

You are talking about a shrinkage estimator. Imdb is possibly the most famous example of this, how they calculate which movies will make it onto the top 250. It relies on the equation,

weighted rating (WR) = (v ÷ (v+m)) × R + (m ÷ (v+m)) × C , where:

* R = average for the movie (mean) = (Rating)
* v = number of votes for the movie = (votes)
* m = minimum votes required to be listed in the Top 250 (currently 3000)
* C = the mean vote across the whole report (currently 6.9)

They call this a "true bayesian rating" and that's true in the sense that our prior for the parameter "average rating" is that it is the same as for all other movies. This prior is then updated based on the "likelihood," which is the average rating for that movie, which has more strength if it has more votes. But I'm not sure whether this technically qualifies as bayesian, because neither the prior nor the posterior is a distribution... Can anyone clarify this?

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you for this answer. I wonder why it doesn't seem to work for me, here is an example: docs.google.com/spreadsheets/d/… Movie 2 should have higher WR based on the number of votes than Movie 3 but it doesn't. What am I getting wrong? $\endgroup$
    – elbatrofmoc
    Commented Sep 30, 2022 at 9:16
7
$\begingroup$

You could use a system like reddit's "best" algorithm for sorting comments:

This algorithm treats the vote count as a statistical sampling of a hypothetical full vote by everyone, much as in an opinion poll. It uses this to calculate the 95% confidence score for the comment. That is, it gives the comment a provisional ranking that it is 95% sure it will get to. The more votes, the closer the 95% confidence score gets to the actual score

So in the case of 3 people voting 5/5, you might be 95% sure the "actual" rating is at least a 1, whereas in the case of 500 people voting you might be 95% sure the "actual" rating is at least a 4/5.

Improve this answer
$\endgroup$
1
$\begingroup$

You may run into problems implied by the Gibbard Satterthwaite Theorem or Arrow's Impossibility Theorem, or any of the results of voting theory...

Improve this answer
$\endgroup$
1
$\begingroup$

There is a simple (also to implement) heuristic to first seed the pool of votes with small number of dummy votes with average voting, and later replace it with incoming votes.
So, for instance, new object appears and you give it few votes rating it 2.5/5 (this is the best you can tell about it at the zero-knowledge point). Then the first vote comes, let's say 5/5, but it is somewhat tempered by the rest of the initial pool and the objects mean is slightly above 2.5. Than next votes come and the mean gradually moves from the initial guess to the real average which than has time to stabilize. Finally this algorithm converges to normal vote mean.

Improve this answer
$\endgroup$
1
  • $\begingroup$ "...and later replace it with incoming votes." Just a little fix. In 'standard' bayesian average dummy votes are permanently added (and not replaced by actual votes), so (providing - as in your example - that only 5/5 votes are given) the value converges but it will never show the exact average. $\endgroup$
    – Tomaso Neri
    Commented Dec 3, 2015 at 10:12
1
$\begingroup$

You can use a Bayesian approach.

If you have no votes at all then a naïve rating for an item would be based on the distribution of ratings/votes among all the items. That would be the prior distribution for the rating

Then with more data/votes for the item you can update your estimate about the true distribution of ratings/votes for an item and compute an estimate for the rating.

So you have observations of votes that are categorical $1,2,3,4,5$. You could describe the prior for this by a Dirichlet distribution (whose parameters are estimated based on the items that you already know). Then the posterior will also be a Dirichlet distribution (since the Dirichlet distribution is the conjugate prior).

Improve this answer
$\endgroup$
0
$\begingroup$

You can choose the lower bound of a $1-\alpha$ confidence interval for a binomial proportion, i.e. Clopper-Pearson interval. Or, if you need a closed formula, you can use the lower bound of the Wilson interval, i.e. $$\frac{1}{1+z^2/n}\left[\hat{p} + \frac{z^2}{2n} - z \sqrt{\frac{\hat{p}(1-\hat{p})}{n}+\frac{z^2}{4n^2}}\right]$$ where $z$ is the $1-\alpha/2$ quantile of the standard normal distribution.

Edit: Sorry, the suggested confidence intervals only make sense for binary votes (like/like not). But the underlying idea also works with other confidence intervals: the lower bound will be larger for larger numbers of votes.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.