What is the point of using a Bayesian prior?

Question

I do struggle with the most basic starting point of Bayesian statistics: why is using a prior useful?

It seems to me that if anything they hurt much more than help. Moreover, Bayesians always say things like "the more evidence you get the less important a prior becomes". So why use them in the first place?

Especially if you have start with a prior that is really far-off, then you will be hurting your estimates.

To me a "Frequentist" approach seems much simpler and more straightforward.

I would like to discuss here an example that is very typical in introductory Bayesian course/explanations (e.g. this, this, or this)

Example 1 -- Are French people Rude?

Imagine that I am interested in estimating if French people are rude (in a binary way: Rude vs Non-Rude).

Imagine that the true parameter $\theta$ is 0.3, 30% of French are rude.

I have a random sample of 100 people with data on "rudeness".

A straightforward "Frequentist" approach would be: compute your confidence intervals on the sample and ... you done. We know what is the probability that the true $\theta$ lies within our CI, and we know that our sample average, on average, thanks to the Central Limit Theorem, will be close to the true parameter.

    true_theta = 0.3
    set.seed(111)

    # population
    X = rbinom(n = 10000, size = 1, prob = true_theta)

    # sample of 100
    x = sample(X, size = 100)
    p_hat = prop.table(table(x))[2]
    margin_error = 1.96 * sqrt( (p_hat*(1-p_hat)) / 100  
    ci_low = p_hat - margin_error
    ci_high = p_hat + margin_error
    data.frame(p_hat, ci_low, ci_high)

Now Bayesian will try to incorporate priors into this. Why?

We can imagine that most people will have beliefs that French are rude. (Using a Beta distribution) based on personal experience, someone would typically have a prior distribution like this: they know 10 French people and 7 are rude. So that is their prior.

The results are shown in the Figure above. On the bottom panel, I draw the Confidence Intervals around the sample mean.

It seems to me that the Frequentist approach will gives us a more precise and immediate answer to this question.

What am I missing here?

Example 2 -- Forecast Knock Out

Imagine I am interested in forecasting Knocked out (KO) in the Ultimate Fighting Championship (UFC).

My basic Frequentist approach would go like this.

Study the sport and see what variables play a role in predicting a fighter knocking another fighter (e.g. age of the fighters, winning streak, home advantage, …).

Then I will sample 30 UFC events and start build my model. I would use a simple logistic regression with maximum likelihood. I don’t have prior per see, but I have a model of reality that I built on theoretical knowledge, like any scientists do, Bayesian or not. My initial model would be that age and style of fighting (Muay Thai fighter vs BJJ) would be the most important factors predicting a KO.

I retrieve estimates from my logistic regression. Then I would cross-validate, take another sample and apply my initial model to the new data and see how it perform.

Let’s imagine that it doesn’t perform too well. I will then try to improve my model in studying more closely the sport. I find that accounting for an interaction between injury and age of the fighter is one of the most important predictors a KO. I re-run my regression with this and again cross validate, and now the model perform well.

Now I have a forecasting model telling me the probability of a KO and I can use it to bet money.

Furthermore, I know that certain special events, like a New Year's Eve event, will have more KOs, so I can adjust my model according to that fact.

Where would a prior, in the Bayesian sense, be useful here? Frequentists like all scientists use model to test reality and improve their models over time, but the difference is that they don’t put a formal initial probability on things. I still do not see why, you would want to do that.

    # code for the Figure
    theta_range <- seq(0, 1, by = 0.01)
    prior <- dbeta(x = theta_range, shape1 = 7, shape2 = 3)

    # observed success
    success_obs = table(x)[2]
    failure_obs = table(x)[1]

    # sampling distribution #
    likelihood <- dbinom(x = success_obs, size = 100, 
                           prob = theta_range) * 100

    # posterior 
    posterior_density <- likelihood * prior

   df <- data.frame(theta = theta_range,
                     likelihood = likelihood,
                     prior_dist = prior,
                     posterior_distribution = posterior_density)

    df_melt = melt(df, id.vars = 'theta')

    df_melt %>%
      ggplot(aes(x = theta, y = value, color = variable)) + 
      geom_line(size = 1.2) + 
      scale_x_continuous(breaks = seq(0, 1, by = 0.1)) + 
      geom_vline(xintercept = p_hat, size = 1.5) + 
      geom_vline(xintercept = ci_low, colour = 'gray', 
                                      size = 1.5) +
      geom_vline(xintercept = ci_high, colour = 'gray', 
                             size = 1.5) + theme_minimal() +
      ggtitle("Bayesian + Frequentist Confidence Intervals")

Answer 1

There are many excellent thoughts here. There is a short answer to the question. If you want to gauge relative evidence you can sometimes get away without using any outside information. Relative evidence can be summarized by a likelihood ratio in the likelihoodist school of statistics. For example one may use study data to compute the likelihood ratio assuming that true mean blood pressure is 120mmHg vs. the mean being 140mmHg. Or you can quantify evidence more indirectly using p-values (evidence against something, only). If you want to quantify absolute evidence there is no mathematical way to compute "absolute" probabilities without having a prior distribution. So if you wanted to compute the probability that the true mean blood pressure is between 135mmHg and 145mmHg you would need a prior. Likewise if you wanted to compute the probability that a medical treatment lowers mortality instead of just using a frequentist hypothesis test to compute the probability of getting data stranger than ours if the treatment does nothing about disease risk you would need a prior.

The most compelling stories I've seen about the Bayesian approach are Nate Silver's The Signal and the Noise and Bernoulli's Fallacy by Aubry Clayton. More thoughts are here and here.

An analogy in medical diagnostic testing is often useful. Sensitivity and specificity are sometimes used as test characteristics. These condition on the actual disease status so only provide relative information. To turn them into absolute information (probability of disease) one must use Bayes' rule to factor in disease prevalence (the prior).

So the prior distribution is the mathematical cost of being able to make direct statements that are not just about relative evidence.

Answer 2

Questions

why is using a prior useful?

How do you "not use a prior"? Even the frequentist approach has a prior—it's just unstated, but it's there all the same. There's always a prior, whether you like it or not.

As for which prior you should use, that's a different question, which other answers on this site go over. See for example this question.

Especially if you have start with a prior that is really far-off, then you will be hurting your estimates.

I know what you're trying to say here, but it doesn't really make any sense. For your prior to not "hurt your estimates", it should reflect your data. But that's not the goal of the prior—that's the goal of the posterior. The prior should reflect any prior knowledge you have, and otherwise be "uninformative" (if you believe that priors can be uninformative, which I do).

Example 1

You can verify that your frequentist approach does have a hidden prior by simply redoing your calculation after transforming the parameter $\theta$. For example, let $\beta = e^\theta$. The frequentist approach produces a different distribution and confidence interval. The parametrization is part of the unstated prior.

Example 2

Saying that you're going to use maximum likelihood to make your estimate is identical to using MAP with a flat prior. There is nevertheless a prior, which is dependent on the model you choose. As before, this includes its parametrization. You can examine your prior by evaluating estimates before updating parameters given new data.

Answer 3

Here's a example of how to use Bayesian priors in a way that even "frequentists" agree is useful.

Let's say you want to estimate the know how well students at 100 different schools are doing in math, so you can identify schools that are doing particularly well or poorly. But you can only assess math knowledge through a test that not all students took. At most schools well over 100 students take the test but at some schools only a few take it due (assume nonresponse is random). Overall, the average score on test is 85% but obviously you are more interested in the average score at particular schools.

Now let's say that at school B only 5 students took the test and their average score was 50%. What are we to make of this?

A purely frequentist approach would to take the data at it's word and treat 50% as the best estimate available of the average score at that school, with some fairly large confidence intervals of course.

But that seems problematic. We know that overall the average test score across all students is an 85%. Given that we only have data from five students at this school, doesn't it seem MORE likely that the true value is actually somewhere closer to the 85% than 50%? The Bayesian approach to this problem would be to treat the overall mean as a "prior" and then update that prior with the additional data we got from the five students at this school. This is going to "shrink" our final estimate towards the mean by some amount. Since we only have five students at this school it's going to shrink it by quite a bit, since the data are weak. At a school with 150 respondents we would put more trust in the data and only shrink a little bit.

This approach is called "empirical Bayes estimation" and it's widely used in multilevel modeling, even by people who don't think of themselves as Bayesian, and more explicitly Bayesian versions of this approach (Google "Multilevel regression with post-stratification") are very common in political science when trying to get estimates of public opinion in small states

This is in fact what Fivethirtyeight.com does to predict elections in the US. To estimate the chance that a given candidate (say Trump) will win a state they look at polling at that state, but then they "shrink" the result of that polling towards a prior that they got from other data.

For example, let's say that before the 2020 campaign even starts you decide that, based on demographic trends, partisan affiliation and presidential approval numbers, Trump is only likely to get 30% of the vote in Vermont (this is our prior). Then someone does a poll of 100 people and finds that Trump is actually winning in Vermont with 51% of the vote. A frequentist would have to either put total trust in this result or ignore it completely. A Bayesian can do something more subtle: we use Bayes' rule to shrink this estimate towards our prior by some amount. In other words we don't actually believe that this poll means that Trump is actually ahead in Vermont, but we also no longer totally believe our old prior that Trump was only going to get 30%. Maybe now we think that we will win 40% of the vote. Then when we get even more new data, we update the new prior again.

Philosophical disagreements aside, this approach really works, which is why Fivethirthyeights's forecasts are so accurate: https://projects.fivethirtyeight.com/checking-our-work/

Answer 4

So an important area of work that I am involved in is decision making under uncertainty, particularly where money is placed at risk. You are asking the wrong question.

The first question should be when and where should I use a Bayesian method and when should I not, maybe never, use a Bayesian method. The subsidiary question would be about why you care about the answer. An academic and a person solving an applied problem, even if the question is identical, probably should not use the same method.

Bayesian methods are good at helping me understand my beliefs and change them as new information comes in. That me could be an us and the my could be an our. If you have no beliefs on the topic, why do you even need to consider the topic?

Frequentist and Maximum Likelihood methods work really well when we lack prior belief because they minimize the maximum amount of risk that you will be facing. Indeed, any two people that get the same result on a t-test with the same cutoff criterion should infer the same thing and decide the same thing. That should not be true with Bayesian methods.

Now let us get back to my area of expertise. How should I gamble money?

Well, it turns out that there are two very nice binding sets of rules that govern my behavior when money is involved. The first is the Dutch Book Theorem. If I were to massively oversimplify it, it says that if you gamble, you can never use a non-Bayesian method or it is possible to force you to take a sure loss. In some cases, I can force you to take a sure loss 100% of the time.

If you are in finance, as I am, methods such as ordinary least squares can guarantee a loss to the user. I do some interesting training exercises to show how that is done. You get some seriously stunned looks when they lose and realized they could never have won. Simple rule, you cannot use non-Bayesian methods when placing money at risk in a gambling or market type of situation.

The second rule comes because the Dutch Book Theorem and its converse imply that it is necessary to use a Bayesian method but it doesn't say that it is sufficient. It turns out that it is not a sufficient criterion.

The second rule requires the use of a proper prior built from actual information. The reason is that all admissible Frequentist rules are either the limiting form of a Bayesian rule or match the Bayesian outcome in every sample. That leads to the question of whether or not the Frequentist solution inherits the Bayesian blessing when gambling, or if there are limits on Bayesian gambling.

The result is that there are limits on Bayesian gambling. A market maker or bookie that is clueless about the game they are playing can be forced to take losses by an informed actor. Unfortunately, I don't have citations where I am at. I print things out and put them in binders and I am not near my binders. Nonetheless, if you did an academic search on the Dutch Book Theorem, you would likely come across them pretty quickly.

Your question is about the obvious problem of "what if the prior is wrong?" Well, it can never be wrong because, for it to be valid, it should reflect your beliefs. It isn't that the prior is wrong, it is that your beliefs are wrong. The data should update your beliefs. If you are very prejudiced, then they may be updated by an almost imperceptibly small amount.

Remember, in the Frequentist world, $\theta$ is a fixed point. There is a correct single answer. In the Bayesian world, $\theta$ is a random variable. Even if there is a fixed, correct answer, it is quite likely that you will never find it. That is okay, you have your distribution and if it is in the near neighborhood of reality, then all is well.

You cannot pass COVID on to others because COVID does not exist, therefore you do not need to isolate or quarantine if somebody tells you that you are exposed.

That is a belief. It is wrong, but it is a belief. Given the right type of data, you may interfere with or alter that belief. Bayesian methods begin with you and where you are at.

How do you know that your sample of French people was representative? Could you not have acquired a pleasant and welcoming subsample by chance? Why trust the data alone? Why not, instead, question if you are prejudiced?

Answer 5

From whatever little I know about Bayesian method, the best use case is when you have limited pieces of evidence which by themselves are not enough to make a frequentist probability with reliability. But the little evidence you have can add to an informative prior probability to take you closer to the truth.

In the particular example you provided, I don't see a great use case for Bayesian application, but consider how you can solve this problem without using Bayesian interpretation - question from arbital

Suppose there's a bathtub full of coins: Half the coins are "fair" and have a 50% probability of coming up Heads each time they are thrown. A third of the coins are biased to produce Heads 25% of the time (Tails 75%). The remaining sixth of the coins are biased to produce Heads 75% of the time. You randomly draw a coin, flip it three times, and get the result HTH. What's the chance this is a fair coin?

Would it be doable if you throw away the prior information and see how you can answer the question maybe rephrased as "what is the probability of heads for a single toss of this coin which gave a HTH"?

Answer 6

A prior is useful for incorporating historical information, analogous to a fixed effect meta-analysis. Because Bayesians define parameters as random variables and probability as the belief of the experimenter they may feel much freer to incorporate information from other studies or their personal beliefs. In contrast, because frequentists are concerned with the performance of their testing procedure they may scrutinize the compatibility of studies. The frequentist needs to feel comfortable assuming the unknown fixed true quantity being investigated in each study is identical (even if the subject-level observations are not necessarily exchangeable). This may be a big assumption to make, analgous to assuming an informative Bayesian prior. Using a purely subjective Bayesian prior is analogous to performing a meta-analysis with hypothetical experimental data.

I will argue against using a Bayesian prior and in favor of a frequentist meta-analysis, beginning with an example taken from Decision Making in Drug Development via Confidence Distributions (Johnson 2021). The primary reason not to use a Bayesian prior is that a subjective interpretation of probability as a measure of belief is unfalsifiable (1)(2).

A confidence distribution $H(\theta,\boldsymbol{x})$ as a function of the hypothesis and observed data has the appearance of a cdf on the parameter space and depicts p-values and confidence intervals of all levels based on a particular testing procedure. This same information can be displayed as a confidence density and a confidence curve.

The figure below depicts a meta-analysis using confidence distributions for a binomial proportion $\theta$. Density (a) represents an informative prior distribution based on historical data and a vague conjugate prior with an estimate of 0.90 and a sample size of n = 50. This same information is depicted in (b) as a confidence density resulting from a likelihood ratio test. A similar confidence density can be produced by inverting a Wald test with a logit link. The posterior based on the current data binomial likelihood and a vague conjugate prior is shown in (c) with an estimate of 0.87 resulting from n = 30. This same information can be represented as a likelihood ratio confidence density, (d). Using Bayes theorem, (a) and (c) combine to form (e). Multipliying the historical and current likelihoods and inverting a likelihood ratio test forms (f). This multiplication of independent likelihoods is precisely what Bayes theorem accomplishes (plus normalization), without the inversion of a hypothesis test.

The confidence densities above can be interpreted under a Neyman-Pearson framework using a pre-specified null hypothesis and type I error rate. they can also be interpreted under a Fisherian framework of evidential p-values to compare the plausibility of multiple hypotheses, not necessarily pre-specified. Many Bayesians will use a Dutch book argument in support of posterior probability. Here is an example where the long-run characteristics of a likelihood ratio test are used to form a Dutch book against two different Bayesian posteriors. (3)