Sorting answers, given overvotes and undervotes

Question

At many question-and-answer sites, like StackExchange, people can upvote or downvote each answer. These sites also typically try to use the votes to sort answers, so the answers that are most likely to be helpful or accurate tend to appear nearest the top. Given the number of upvotes and downvotes for each answer, how should one sort the answers?

---

Examples. To illustrate why I think this question might have some non-trivial statistical content, let me survey a handful of methods one might consider and some shortcomings of them. In each method, we compute a score for each answer from its upvotes and downvotes, and then sort the answers by score, so the only question is what method to use to compute the score.

• Upvotes minus downvotes. The additive difference is simple to compute. Limitations. Is an answer with 121 upvotes and 100 downvotes really better than an answer with 20 upvotes and 0 downvotes?

• Upvotes divided by total number of votes. This estimates the fraction of voters who held a positive opinion of the answer, which is arguably natural and meaningful in its own right. Limitations. Is an answer with 1 upvote and 0 downvotes really better than an answer with 8 upvotes and 1 downvotes?

• Compute a confidence interval. I suppose we could compute, for each answer, a confidence interval for the true fraction of people who would vote positively if they voted. But it is not clear how to extend this to a complete sorting scheme; what do we do when two answers have overlapping confidence intervals?

• Laplace smoothing. We could apply Laplace smoothing (additive smoothing) to the counts of upvotes and downvotes, then estimate the fraction of votes that are positive. If we have $u$ upvotes and $d$ downvotes for an answer, its score will be $(u+1)/(u+d+2)$. (For instance, this would declare that an answer with 3 upvotes and 1 downvote is equivalent an answer with 1 upvote and 0 downvotes. Does that seem reasonable? Hard to say.)

Answer 1

I recently learned about Bayesian rating. Bayesian rating computes a weighted average of the naive uncorrected rating for this answer and the average rating for all answers. The fewer the votes for this answer, the more we weight things towards the average rating. The intuition is that if we have no votes on this answer, then the average rating (across all answers) is our best guess at the rating of this answer; as we gain more votes on this answer, they start to move our estimate away from the average rating.

The scheme apparently works like this. The naive uncorrected rating for an answer is $r = u/n$, where $u$ is the number of upvotes and $n$ the total number of votes for that answer (i.e., $n=u+d$, where $d$ is the number of downvotes). Let $r^*$ denote the average uncorrected rating, averaged over all answers, i.e., the average of $r$ over all answers. Also, let $n^*$ the average number of votes per answer, over all answers, i.e., the average value of $n$.

With these definitions, the Bayesian rating for an answer is $r'$, defined as follows: $$ r' = \frac{n}{n+n^*} r + \frac{n^*}{n+n^*} r^*. $$ Notice how if we have no votes on this answer, then $r'=r^*$ (the Bayesian rating is the average rating across all answers); whereas as $n \to \infty$, we have $r' \to r$ (the Bayesian rating for an answer converges to the naive uncorrected rating, as the number of votes on the answer becomes large). These seem like appealing properties.

For a generalization that can be used if items can be rated from 1-5 stars, instead of upvoted/downvoted, see the answers to How to find confidence intervals for ratings?, the essay Bayesian sorting by rank (which appears to use a slightly different formula), and Bayesian Ratings: Your Salvation for User-Generated Content.

Answer 2

I found an essay (How Not To Sort By Average Rating) which argues that you should compute a 95% confidence interval for the true fraction of people who would vote positively, and then sort ratings by the lower end of the confidence interval.

This is an interesting approach. It is hard for me to judge whether this is in some sense the "right" way to do it, but it seems to behave better than some of the more naive alternatives. Worth noting: @raegtin gives a cogent criticism of this approach, in his answer to a related question. The short version of the criticism is that using the lower limit of a confidence interval pushes items with few ratings towards a very low rating, where it's probably better to push them towards the average rating over all items (as Bayesian rating does).