7
$\begingroup$

A Bayesian estimator as defined in the Wikipedia article Practical example of Bayes estimators balances the prior knowledge of the entire data set with the knowledge of the subset. This is usually used when we have a small sample from the subset.

What is a good weight choice for the priori knowledge constant for a Bayesian Estimator?

For example, let's say we have set of restaurants. Those restaurants can be liked or disliked. If we treat "likes" as 1 and "dislikes" as 0 (and clicking on like or dislike as a vote), then we can treat the likability of restaurants as a Bernoulli trial.

For example, let's say for all of the restaurants in the country the average "likes"/votes is 0.7 or 70%.

Now a new restaurant opens up. It is a burger joint and 1 person clicks "like". Should that restaurant get a rating of 100% and immediately jump to the top of the best foodies list? Definitely not. There is only 1 vote.

A way to handle this is with a Weighted arithmetic mean:

w = (m * national_average + restaurant_votes * restaurant_average) / ( m + restaurant_votes)

Doing the math, we get:

(4 * 0.7 + 1 * 1.0) / (4 + 1) = 0.76

So the new burger joint gets a rating of 76% likability.

But what should the value of m be? Is 4 a good choice?

Restaurant comparision Is the El Torito place really better than the Star of India??

If one treats each star rating as up to five likes, then the above applies.

Looking at the Wikipedia article Practical example of Bayes estimators, it gives this example from IMDB and looking back in 2012, the constant of m was chosen to be 3000. Why 3000?

Given the above formula what is a good weight value for m?

The Naive Bayes spam filtering: Dealing with rare words article suggests a value of 3 is a good value if it is a random variable with beta distribution.

The Agresti-Coull Interval hints at a choice of the prior knowledge of z^2 for 3.8416 or essentially 4 given the rule of thumb "add 2 successes and 2 failures".

Is this really a Bayesian estimators question? Looking at this Bayes' Estimators, the formulas look a lot more complex...

Update: This paper adds insight to the choice of weight: TO THE BASICS: BAYESIAN INFERENCE ON A BINOMIAL PROPORTION It relates to a level of certainty.

References:

Agresti-Coull Interval

Practical example of Bayes estimators

Naive Bayes spam filtering: Dealing with rare words

$\endgroup$
  • $\begingroup$ Your examples of Naive Bayes spam filtering and Agresti-Coull interval are not related to this problem. In the first case you make binary classifications, in the second we are talking about binomial distribution of $k$ successes in $n$ trials -- neither of the cases is even close to problem of assigning pooled ratings based on individual ratings... $\endgroup$ – Tim Jul 30 '16 at 15:16
  • $\begingroup$ Let's let others comment as well. researchgate.net/publication/… suggests that Agresti-Coull is a Bayesian estimator. $\endgroup$ – Chris Jul 30 '16 at 15:34
  • $\begingroup$ Yes, but it estimates a different thing! Ratings do not follow binomial distribution. $\endgroup$ – Tim Jul 30 '16 at 15:37
3
$\begingroup$

I'm providing second answer since it is either: problem formulation that is unclear, or the answer provided by OP is wrong, since it does not address the problem. In my answer I'll try to refer to both of the cases.

First, let's try to define the problem. You have rankings of restaurants based on votes, where each vote is either "like" coded as $1$, or "dislike" coded as $0$. This means that we are dealing with Bernoulli distributed random variable. If you count the number of "likes", you have binomial distribution with $k_i$ likes per $n_i$ votes, for $i$-th restaurant. You are interested in the probability of restaurant being "good", $\theta_i$. The simple estimate of $\theta_i$ is $k_i/n_i$ (likes/votes), but as you already noticed, this does not account for the fact then restaurants differ in the number of votes they got, so some rankings are more reliable than others.

This problem may be formulated in terms of beta-binomial model, where we use conjugate beta prior for binomial likelihood function. In such case we define our model as follows

$$ \theta_i \sim \mathrm{Beta}(\alpha, \beta) $$ $$ k_i \sim \mathrm{Binomial}(n_i, \theta_i) $$

so we assume beta prior for $\theta_i$ parametrized by $\alpha$ and $\beta$. This is a Bayesian model, so you can recall that Bayesian model is defined in terms of likelihood and prior, that both taken together tell you about posterior probability of your parameter given the data and priors

$$ \color{violet}{\text{posterior}} \propto \color{red}{\text{prior}} \times \color{lightblue}{\text{likelihood}} $$

This means that the prior information you include in your model may influence the results, however the more information your data contain (relative to the prior), the more likely it is going to overcome the information contained in the prior.

So choosing a prior means making a subjective decision that can possibly affect your model (this is why Bayesian approach was criticized by some). Of course, you can make such choice of prior that brings as little as possible information into the model and let's "the data talk", i.e. weekly informative prior (there is no such a thing as "uninformative" prior). In case of beta-binomial model, you can choose for that beta distribution with parameters $\alpha = \beta = 1$, that leads to uniform prior. This means that you assume that $\theta_i$ can be any value between $0$ and $1$ with equal probability. Such assumption does not seem to bring much subjectivity into the model, but notice that what follows is that you assume a priori that $\theta_i$ has mean

$$ \frac{\alpha}{\alpha+\beta} = \frac{1}{1+1} = 0.5 $$

since this is the mean of $\mathrm{Beta}(1, 1)$ distribution. So if you have no data at all, then you "estimate" the ranking to be $0.5$.

Until now, we had no data for discussing this question so let me make up some data. Say that in your database you have in total $N=53480$ votes, where $K=34561$ are "likes" ($65\%$). As examples I'll use three restaurants:

# likes votes
1     1     1
2     3     4
3    19    25

Under beta prior the posterior mean is

$$ \frac{\alpha + k_i}{\alpha+ k_i + \beta + n_i - k_i} = \frac{\alpha + k_i}{\alpha + \beta + n_i} $$

So under $\alpha = \beta = 1$ parameters you would estimate posterior means $\bar \theta_1 = 0.66$, $\bar \theta_2 = 0.66$, and $\bar \theta_3 = 0.74$ (blue lines on the plots below, where violet lines mark simple estimates $k_i/n_i$). You can notice when we do not have much data (much information), the posterior means are shrinked towards the prior means.

Posterior distributions of parameters

You may be however interested in using informative prior, i.e. bringing some out-of-data information into your model. One such choice would be to center your beta distribution on global mean, with $\alpha$ and $\beta$ chosen in proportionally to how much you want to insist on your prior mean (how strongly would your prior shrink posterior towards it), as in the link that you posted. The more informative you make your model, the more influence it would have on your results. Unfortunately, since the final result depends on both your data and the prior, there is no single valid choice for the parameters, since they will always be problem-specific. On the plot below you can see different such choices.

Beta distribution with different parameter values

You may think of setting prior mean to $K/N$ (global mean) and sample size to $N-K$ (the sample values calculated as in the link you posted), but with choosing such prior you would need more data then is in the whole database to make your posterior estimate close to the arithmetic mean and this does not sound reasonable.

In both cases (weekly informative and informative priors), you would end up with totally valid Bayesian estimates (in fact, "handbook" examples), but the choice of $\alpha$ and $\beta$ is subjective and even if you decide for a weekly informative prior, so you still bring some a priori information in your model.

While this approach "works", there are few problems connected to your needs as described in the question:

  • It does not account for the fact that restaurants differ in the number of votes, so it does not correct for their reliability. When using weekly informative prior $\alpha = \beta = 1$, for very small counts of votes the results will be influenced by prior and shrinked towards $0.5$, but that is all. When using informative prior, results will be shrinked towards the prior mean, but this leads to further complications (see below).
  • While in case of IMBD estimator you need to specify single parameter $m$, in case of beta-binomial model, you need to decide about two parameters. This does not seem to simplify your problem. Of course, you can re-define beta distribution to be parametrized by mean and sample size (or precision), as in the link you posted, but this still does not help with the fact that you need to make a subjective choice about it. In fact, choosing $m$ for the IMDB estimator is also about how many votes you consider as reliable, so it is also about quantifying your certainty.
  • In fact, in case of IMDB estimator the choice of $m$ parameter is more obvious since it tells you simply that one vote counts as $1/m$ pseudo-votes equal to global mean, what makes deciding about the parameter actually easier.
  • Finally, choosing $\alpha$ and $\beta$ parameters in beta-binomial model does not help you anyhow in choosing the $m$ parameter in the IMDB estimator since both methods work differently.

So while there is no reason why choosing beta-binomial model would be a bad choice, it does not solve the the problem of deciding about the parameter.


Briefly commenting on other choices you considered:

  • Adding two successes and two failures as in Agresti-Coull estimator for confidence intervals does not differ that much from the beta-binomial model described above.
  • In the Wikipedia page about naive Bayes spam filtering they mention adding 3 to the results calling it "good value", but they do not provide any reference for that suggestion and any rationale behind it, so I do not see any reason for treating it seriously. I guess it is connected to smoothing the data that is often done when working with language data, but I don't think it relates to your problem.
  • Using $m=30$ because sample size of $30$ was described in old textbooks as a rule of thumb for central limit theorem is not a good choice. First, the choice of $30$ was pretty arbitrary. Second, central limit theorem says nothing about "goodness" of the data (check What intuitive explanation is there for the central limit theorem?).
  • You ask why IMDB used $m=3000$. I guess they decided on it either by observing something like "80% of the movies have number of votes above it", or by making research that has shown that this value is optimal (e.g. makes their rankings correlate with some external criteria, as described in my first answer).
$\endgroup$
1
$\begingroup$

The IMDB ratings example is a simple one, so you are right that in many cases they get more complicated.

Actually, the answer to your question is already given in the Wikipedia entry:

Note that $W$ is just the weighted arithmetic mean of $R$ and $C$ with weight vector $(v, m)$. As the number of ratings surpasses $m$, the confidence of the average rating surpasses the confidence of the prior knowledge, and the weighted bayesian rating ($W$) approaches a straight average ($R$). The closer $v$ (the number of ratings for the film) is to zero, the closer $W$ gets to $C$, where $W$ is the weighted rating and $C$ is the average rating of all films. So, in simpler terms, films with very few ratings/votes will have a rating weighted towards the average across all films, while films with many ratings/votes will have a rating weighted towards its average rating.

So in the formula

$$ W = {Rv + Cm\over v+m} $$

$m$ is prior information or belief. Saying it in plain English: by default the movies are rated closer to overall average of votes for all of the movies $C$, but as they grab more votes, they get closer and closer to average of their votes $R$. By specifying $m$ you decide about the threshold. There is no "good", or "bad" choices since it is a subjective choice based on how heavy you want the overall average $C$ to weight on your estimate. You simply state that one real vote counts as $1/m$ pseudo-votes based on overall average. It is called prior because it is based on a priori, out-of-data information -- it may be based on your beliefs, some previous experiments that suggest so, expert opinions, etc. It is one of the core aspects of Bayesian approach that you include such out-of-data information in your model. There is simply no objective criterion saying something like "4 ratings are not trustworthy, but 5 are", stating something like this would be ridiculous, so you are forced to make some arbitrary decision on it.

This estimator is designed to have threshold that can be flexibly adapted to your needs. For example, if your site has on average 1000 ratings per item, you do not want to threat items with 5, 10, or even 100 ratings as equally reliable as the majority.

Of course, you can always try to find optimal $m$ that maximizes some optimality criterion, e.g. correlation of ratings with movie theater's ticket sales, but then it would be hard to call it prior since it'd be just some unknown parameter to maximize. Then it also wouldn't be a Bayesian estimator. If the subjective aspect of this method does not appeal to you, you can always use arithmetic mean of the ratings, i.e. $m=0$. This does not use any arbitrary threshold.

$\endgroup$
  • $\begingroup$ But surely it's not purely subjective. For example, central limit theory suggests that when a sample size approaches 30, it's a pretty good size and thus conclusions drawn will be fairly accurate. Does that suggest an m of 30?? $\endgroup$ – Chris Jul 30 '16 at 7:48
  • $\begingroup$ @Chris no, it is purely subjective. Say you have rating site where no movie gets more then 5 votes - if you chosen m=30 then you would estimate average rating for all of the movies. Imagine that you have site where almost all of the movies get at least 1000 votes - then m=30 would not change anything about the ratings and you could get rid of it... $\endgroup$ – Tim Jul 30 '16 at 8:08
  • 2
    $\begingroup$ Besides, no theory states that 30 is "big enough" sample for anything. $\endgroup$ – Tim Jul 30 '16 at 8:09
  • $\begingroup$ "The rule of thumb is that a sample size n of at least 30 will usually suffice...' math.uah.edu/stat/sample/CLT.html $\endgroup$ – Chris Jul 30 '16 at 14:00
  • $\begingroup$ The MN spam article mentions the Beta distribution implying that is the justification for choosing 3. $\endgroup$ – Chris Jul 30 '16 at 14:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.