# What is the point of using a Bayesian prior?

I do struggle with the most basic starting point of Bayesian statistics: why is using a prior useful?

It seems to me that if anything they hurt much more than help. Moreover, Bayesians always say things like "the more evidence you get the less important a prior becomes". So why use them in the first place?

Especially if you have start with a prior that is really far-off, then you will be hurting your estimates.

To me a "Frequentist" approach seems much simpler and more straightforward.

I would like to discuss here an example that is very typical in introductory Bayesian course/explanations (e.g. this, this, or this)

### Example 1 -- Are French people Rude? Imagine that I am interested in estimating if French people are rude (in a binary way: Rude vs Non-Rude).

Imagine that the true parameter $$\theta$$ is 0.3, 30% of French are rude.

I have a random sample of 100 people with data on "rudeness".

A straightforward "Frequentist" approach would be: compute your confidence intervals on the sample and ... you done. We know what is the probability that the true $$\theta$$ lies within our CI, and we know that our sample average, on average, thanks to the Central Limit Theorem, will be close to the true parameter.

    true_theta = 0.3
set.seed(111)

# population
X = rbinom(n = 10000, size = 1, prob = true_theta)

# sample of 100
x = sample(X, size = 100)
p_hat = prop.table(table(x))
margin_error = 1.96 * sqrt( (p_hat*(1-p_hat)) / 100
ci_low = p_hat - margin_error
ci_high = p_hat + margin_error
data.frame(p_hat, ci_low, ci_high)


Now Bayesian will try to incorporate priors into this. Why?

We can imagine that most people will have beliefs that French are rude. (Using a Beta distribution) based on personal experience, someone would typically have a prior distribution like this: they know 10 French people and 7 are rude. So that is their prior.

The results are shown in the Figure above. On the bottom panel, I draw the Confidence Intervals around the sample mean.

It seems to me that the Frequentist approach will gives us a more precise and immediate answer to this question.

What am I missing here?

## Example 2 -- Forecast Knock Out

Imagine I am interested in forecasting Knocked out (KO) in the Ultimate Fighting Championship (UFC).

My basic Frequentist approach would go like this.

Study the sport and see what variables play a role in predicting a fighter knocking another fighter (e.g. age of the fighters, winning streak, home advantage, …).

Then I will sample 30 UFC events and start build my model. I would use a simple logistic regression with maximum likelihood. I don’t have prior per see, but I have a model of reality that I built on theoretical knowledge, like any scientists do, Bayesian or not. My initial model would be that age and style of fighting (Muay Thai fighter vs BJJ) would be the most important factors predicting a KO.

I retrieve estimates from my logistic regression. Then I would cross-validate, take another sample and apply my initial model to the new data and see how it perform.

Let’s imagine that it doesn’t perform too well. I will then try to improve my model in studying more closely the sport. I find that accounting for an interaction between injury and age of the fighter is one of the most important predictors a KO. I re-run my regression with this and again cross validate, and now the model perform well.

Now I have a forecasting model telling me the probability of a KO and I can use it to bet money.

Furthermore, I know that certain special events, like a New Year's Eve event, will have more KOs, so I can adjust my model according to that fact.

Where would a prior, in the Bayesian sense, be useful here? Frequentists like all scientists use model to test reality and improve their models over time, but the difference is that they don’t put a formal initial probability on things. I still do not see why, you would want to do that.

    # code for the Figure
theta_range <- seq(0, 1, by = 0.01)
prior <- dbeta(x = theta_range, shape1 = 7, shape2 = 3)

# observed success
success_obs = table(x)
failure_obs = table(x)

# sampling distribution #
likelihood <- dbinom(x = success_obs, size = 100,
prob = theta_range) * 100

# posterior
posterior_density <- likelihood * prior

df <- data.frame(theta = theta_range,
likelihood = likelihood,
prior_dist = prior,
posterior_distribution = posterior_density)

df_melt = melt(df, id.vars = 'theta')

df_melt %>%
ggplot(aes(x = theta, y = value, color = variable)) +
geom_line(size = 1.2) +
scale_x_continuous(breaks = seq(0, 1, by = 0.1)) +
geom_vline(xintercept = p_hat, size = 1.5) +
geom_vline(xintercept = ci_low, colour = 'gray',
size = 1.5) +
geom_vline(xintercept = ci_high, colour = 'gray',
size = 1.5) + theme_minimal() +
ggtitle("Bayesian + Frequentist Confidence Intervals")

• At a high level, a prior is used to incorporate domain knowledge into a problem. To see why domain knowledge is important, see this wonderful answer. Aug 8, 2021 at 17:19
• If you Have a prior then why not use it? Sometimes you do have it genuinely. Aug 8, 2021 at 17:43
• The Bayesian approach can be very natural in sequential/online settings where you have a stream of incoming data. While the initial prior is arbitrary, after some time, your new prior (the previous posterior) will actually encode domain knowledge. On the other hand, if you are estimating a sample mean and have no domain knowledge then I don't really see the utility of using a Bayesian flat-prior approach (unless you plan to use your posterior as a prior in a down stream task).
– user327671
Aug 8, 2021 at 19:51
• @giac hopefully your prior about Paul is that he is very unlikely to be able to psychically predict football results! Aug 9, 2021 at 8:16
• OP, you might find fharrell.com/post/journey writings helpful ( as an experienced frequentist turned bayesian) Aug 9, 2021 at 11:26

There are many excellent thoughts here. There is a short answer to the question. If you want to gauge relative evidence you can sometimes get away without using any outside information. Relative evidence can be summarized by a likelihood ratio in the likelihoodist school of statistics. For example one may use study data to compute the likelihood ratio assuming that true mean blood pressure is 120mmHg vs. the mean being 140mmHg. Or you can quantify evidence more indirectly using p-values (evidence against something, only). If you want to quantify absolute evidence there is no mathematical way to compute "absolute" probabilities without having a prior distribution. So if you wanted to compute the probability that the true mean blood pressure is between 135mmHg and 145mmHg you would need a prior. Likewise if you wanted to compute the probability that a medical treatment lowers mortality instead of just using a frequentist hypothesis test to compute the probability of getting data stranger than ours if the treatment does nothing about disease risk you would need a prior.

The most compelling stories I've seen about the Bayesian approach are Nate Silver's The Signal and the Noise and Bernoulli's Fallacy by Aubry Clayton. More thoughts are here and here.

An analogy in medical diagnostic testing is often useful. Sensitivity and specificity are sometimes used as test characteristics. These condition on the actual disease status so only provide relative information. To turn them into absolute information (probability of disease) one must use Bayes' rule to factor in disease prevalence (the prior).

So the prior distribution is the mathematical cost of being able to make direct statements that are not just about relative evidence.

• Thanks, I will dig into the books you mentioned.
– giac
Aug 9, 2021 at 15:49
• I like this answer, especially the final sentence. Aug 10, 2021 at 10:56

## Questions

why is using a prior useful?

How do you "not use a prior"? Even the frequentist approach has a prior—it's just unstated, but it's there all the same. There's always a prior, whether you like it or not.

As for which prior you should use, that's a different question, which other answers on this site go over. See for example this question.

Especially if you have start with a prior that is really far-off, then you will be hurting your estimates.

I know what you're trying to say here, but it doesn't really make any sense. For your prior to not "hurt your estimates", it should reflect your data. But that's not the goal of the prior—that's the goal of the posterior. The prior should reflect any prior knowledge you have, and otherwise be "uninformative" (if you believe that priors can be uninformative, which I do).

## Example 1

You can verify that your frequentist approach does have a hidden prior by simply redoing your calculation after transforming the parameter $$\theta$$. For example, let $$\beta = e^\theta$$. The frequentist approach produces a different distribution and confidence interval. The parametrization is part of the unstated prior.

## Example 2

Saying that you're going to use maximum likelihood to make your estimate is identical to using MAP with a flat prior. There is nevertheless a prior, which is dependent on the model you choose. As before, this includes its parametrization. You can examine your prior by evaluating estimates before updating parameters given new data.

• "Even the frequentist approach has a prior—it's just unstated, but it's there all the same." Not true. Just because you can write down a frequentist method as a Bayesian method with a certain prior (which doesn't always work, at least not with proper priors) it doesn't mean that "the frequentist approach has a prior". Probability has a different meaning in frequentism, and parameters cannot normally be random variables, so it would be inappropriate to interpret whatever mathematical equivalence exists as if there really were a meaningful Bayesian prior. Aug 9, 2021 at 9:26
• @Lewian Just as you say, "you can write down a frequentist method as a Bayesian method with a certain prior". It doesn't matter how frequentists define probability since the whole point is to view their model through the Bayesian lens under which we agree that they are not avoiding priors. There is always a prior. Finally, I disagree with your idea that "it doesn't always work"—improper priors are still priors. You should write down an example of a frequentist model that "has no prior" to examine it more carefully. Aug 9, 2021 at 9:31
• @Lewian Improper priors are not "a problem". Also, it doesn't make sense to say that you don't use a prior. How can you have a posterior, or a posterior predictive? It sounds like what you mean you say "don't use a prior" is actually use a flat prior. This is still a prior—even if it's improper. Aug 9, 2021 at 9:34
• @Lewian I think I said it clearly in my last comment: "It doesn't matter how frequentists define probability since the whole point is to view their model through the Bayesian lens under which we agree that they are not avoiding priors." Simply changing your definitions is not avoiding priors. By that logic, you could rename prior to slithy toves and claim that you were not using a prior. Clearly, this is nonsense. Aug 9, 2021 at 9:39
• @skan sure, but in my opinion, that's not really a prior. That's extra likelihood from what is essentially additional data. Apr 9 at 0:35

Here's a example of how to use Bayesian priors in a way that even "frequentists" agree is useful.

Let's say you want to estimate the know how well students at 100 different schools are doing in math, so you can identify schools that are doing particularly well or poorly. But you can only assess math knowledge through a test that not all students took. At most schools well over 100 students take the test but at some schools only a few take it due (assume nonresponse is random). Overall, the average score on test is 85% but obviously you are more interested in the average score at particular schools.

Now let's say that at school B only 5 students took the test and their average score was 50%. What are we to make of this?

A purely frequentist approach would to take the data at it's word and treat 50% as the best estimate available of the average score at that school, with some fairly large confidence intervals of course.

But that seems problematic. We know that overall the average test score across all students is an 85%. Given that we only have data from five students at this school, doesn't it seem MORE likely that the true value is actually somewhere closer to the 85% than 50%? The Bayesian approach to this problem would be to treat the overall mean as a "prior" and then update that prior with the additional data we got from the five students at this school. This is going to "shrink" our final estimate towards the mean by some amount. Since we only have five students at this school it's going to shrink it by quite a bit, since the data are weak. At a school with 150 respondents we would put more trust in the data and only shrink a little bit.

This approach is called "empirical Bayes estimation" and it's widely used in multilevel modeling, even by people who don't think of themselves as Bayesian, and more explicitly Bayesian versions of this approach (Google "Multilevel regression with post-stratification") are very common in political science when trying to get estimates of public opinion in small states

This is in fact what Fivethirtyeight.com does to predict elections in the US. To estimate the chance that a given candidate (say Trump) will win a state they look at polling at that state, but then they "shrink" the result of that polling towards a prior that they got from other data.

For example, let's say that before the 2020 campaign even starts you decide that, based on demographic trends, partisan affiliation and presidential approval numbers, Trump is only likely to get 30% of the vote in Vermont (this is our prior). Then someone does a poll of 100 people and finds that Trump is actually winning in Vermont with 51% of the vote. A frequentist would have to either put total trust in this result or ignore it completely. A Bayesian can do something more subtle: we use Bayes' rule to shrink this estimate towards our prior by some amount. In other words we don't actually believe that this poll means that Trump is actually ahead in Vermont, but we also no longer totally believe our old prior that Trump was only going to get 30%. Maybe now we think that we will win 40% of the vote. Then when we get even more new data, we update the new prior again.

Philosophical disagreements aside, this approach really works, which is why Fivethirthyeights's forecasts are so accurate: https://projects.fivethirtyeight.com/checking-our-work/

• Thanks that is useful. In fact, I mentioned post-stratification in one my answers, but I don't "see" it as a prior, rather some kind of "posterior correction". Maybe in the end, you get the same results, but I would rather start with what the data are telling me and then correct if I have a really good reason to do it. For instance, in my forecasting example, I know that some fighters have a X-factor, so I can correct for this. But if that is all what Bayesian is really selling, then it is much more modest than what "Bayesian revolution" claims to be.
– giac
Aug 9, 2021 at 18:54
• You are right that the "post stratification" part of MRP doesn't really have much to do with Bayesianism, and that wasn't what I'm talking about above. What makes MRP and empirical Bayes estimation so popular and successful as methods is their explicit use of priors. Aug 9, 2021 at 19:17
• is Bayesian popular in your field? From my experience in the social sciences, it is still a very niche method.
– giac
Aug 9, 2021 at 20:59
• in political science and sociology it is still fairly niche, but it's growing in popularity among two very different groups: highly sophisticated mathematical stats gurus like Andrew Gelman and more "critical" scholars who think that cultural and social assumptions always influence the way we do stats and so at least we should be honest about it. Both groups agree with what others here have said: we always have some sorts of priors ('biases") when we choose a statistical model and therefore Bayesianism is a more honest approach. Aug 9, 2021 at 21:18
• @giac someone recently asked Andrew Gelman (Bayesian advocate #1) something like your initial question: how do we know a particular method (e.g. Bayesian inference) actually works better than the alternative? So this is something he thinks about a lot. statmodeling.stat.columbia.edu/2021/08/12/… Aug 12, 2021 at 16:11

So an important area of work that I am involved in is decision making under uncertainty, particularly where money is placed at risk. You are asking the wrong question.

The first question should be when and where should I use a Bayesian method and when should I not, maybe never, use a Bayesian method. The subsidiary question would be about why you care about the answer. An academic and a person solving an applied problem, even if the question is identical, probably should not use the same method.

Bayesian methods are good at helping me understand my beliefs and change them as new information comes in. That me could be an us and the my could be an our. If you have no beliefs on the topic, why do you even need to consider the topic?

Frequentist and Maximum Likelihood methods work really well when we lack prior belief because they minimize the maximum amount of risk that you will be facing. Indeed, any two people that get the same result on a t-test with the same cutoff criterion should infer the same thing and decide the same thing. That should not be true with Bayesian methods.

Now let us get back to my area of expertise. How should I gamble money?

Well, it turns out that there are two very nice binding sets of rules that govern my behavior when money is involved. The first is the Dutch Book Theorem. If I were to massively oversimplify it, it says that if you gamble, you can never use a non-Bayesian method or it is possible to force you to take a sure loss. In some cases, I can force you to take a sure loss 100% of the time.

If you are in finance, as I am, methods such as ordinary least squares can guarantee a loss to the user. I do some interesting training exercises to show how that is done. You get some seriously stunned looks when they lose and realized they could never have won. Simple rule, you cannot use non-Bayesian methods when placing money at risk in a gambling or market type of situation.

The second rule comes because the Dutch Book Theorem and its converse imply that it is necessary to use a Bayesian method but it doesn't say that it is sufficient. It turns out that it is not a sufficient criterion.

The second rule requires the use of a proper prior built from actual information. The reason is that all admissible Frequentist rules are either the limiting form of a Bayesian rule or match the Bayesian outcome in every sample. That leads to the question of whether or not the Frequentist solution inherits the Bayesian blessing when gambling, or if there are limits on Bayesian gambling.

The result is that there are limits on Bayesian gambling. A market maker or bookie that is clueless about the game they are playing can be forced to take losses by an informed actor. Unfortunately, I don't have citations where I am at. I print things out and put them in binders and I am not near my binders. Nonetheless, if you did an academic search on the Dutch Book Theorem, you would likely come across them pretty quickly.

Your question is about the obvious problem of "what if the prior is wrong?" Well, it can never be wrong because, for it to be valid, it should reflect your beliefs. It isn't that the prior is wrong, it is that your beliefs are wrong. The data should update your beliefs. If you are very prejudiced, then they may be updated by an almost imperceptibly small amount.

Remember, in the Frequentist world, $$\theta$$ is a fixed point. There is a correct single answer. In the Bayesian world, $$\theta$$ is a random variable. Even if there is a fixed, correct answer, it is quite likely that you will never find it. That is okay, you have your distribution and if it is in the near neighborhood of reality, then all is well.

You cannot pass COVID on to others because COVID does not exist, therefore you do not need to isolate or quarantine if somebody tells you that you are exposed.

That is a belief. It is wrong, but it is a belief. Given the right type of data, you may interfere with or alter that belief. Bayesian methods begin with you and where you are at.

How do you know that your sample of French people was representative? Could you not have acquired a pleasant and welcoming subsample by chance? Why trust the data alone? Why not, instead, question if you are prejudiced?

• "If you gamble, you can never use a non-Bayesian method or it is possible to force you to take a sure loss" - this is not quite true. (a) There may be no real opponent enforcing the sure loss (even the setup of the real situation may not allow anyone to do so). Aug 8, 2021 at 20:53
• (b) There's a real football betting experiment described in P. Walley's "Statistical Reasoning with Imprecise Probabilities" in which the consistent Bayesian had about the worst results of all. People using imprecise probabilities have the option to refuse certain bets (only bet when a bet looks favourable to them), whereas the Bayesian will bet all the time. In real life you don't normally have to bet all the time, so the imprecise probability approach (where you refuse bets that don't look safe enough to you) looks better (and is better in that experiment). Aug 8, 2021 at 20:56
• @Lewian it may be true that there is no real opponent to enforce the sure loss. The Dutch Book Theorem only guarantees that if one exists, then they cannot put you in that position. You are correct. The fact that losses can exist does not imply that anybody noticed them. I will have to look at Walley's article. I will have to understand why the Bayesian must bet all the time as I agree with you. Only bookies, banks, insurers and market makers can be in the position of forced or nearly forced bets. It is why underwriting exists. Aug 8, 2021 at 21:07
• @giac This paper... rss.onlinelibrary.wiley.com/doi/full/10.1111/rssa.12276 ... has something on "falsificationist Bayes", i.e., using Bayesian statistics with a frequentist interpretation of probability (Sec 5, 5.5 in particular). Aug 8, 2021 at 21:37
• @giac, most investers and gamblers would agree that long-run probabilities are the best bet compared to subjective beliefs. I think this is the underlying point of your original question. Aug 9, 2021 at 3:49

From whatever little I know about Bayesian method, the best use case is when you have limited pieces of evidence which by themselves are not enough to make a frequentist probability with reliability. But the little evidence you have can add to an informative prior probability to take you closer to the truth.

In the particular example you provided, I don't see a great use case for Bayesian application, but consider how you can solve this problem without using Bayesian interpretation - question from arbital

Suppose there's a bathtub full of coins: Half the coins are "fair" and have a 50% probability of coming up Heads each time they are thrown. A third of the coins are biased to produce Heads 25% of the time (Tails 75%). The remaining sixth of the coins are biased to produce Heads 75% of the time. You randomly draw a coin, flip it three times, and get the result HTH. What's the chance this is a fair coin?

Would it be doable if you throw away the prior information and see how you can answer the question maybe rephrased as "what is the probability of heads for a single toss of this coin which gave a HTH"?

A prior is useful for incorporating historical information, analogous to a fixed effect meta-analysis. Because Bayesians define parameters as random variables and probability as the belief of the experimenter they may feel much freer to incorporate information from other studies or their personal beliefs. In contrast, because frequentists are concerned with the performance of their testing procedure they may scrutinize the compatibility of studies. The frequentist needs to feel comfortable assuming the unknown fixed true quantity being investigated in each study is identical (even if the subject-level observations are not necessarily exchangeable). This may be a big assumption to make, analgous to assuming an informative Bayesian prior. Using a purely subjective Bayesian prior is analogous to performing a meta-analysis with hypothetical experimental data.

I will argue against using a Bayesian prior and in favor of a frequentist meta-analysis, beginning with an example taken from Decision Making in Drug Development via Confidence Distributions (Johnson 2021). The primary reason not to use a Bayesian prior is that a subjective interpretation of probability as a measure of belief is unfalsifiable (1)(2).

A confidence distribution $$H(\theta,\boldsymbol{x})$$ as a function of the hypothesis and observed data has the appearance of a cdf on the parameter space and depicts p-values and confidence intervals of all levels based on a particular testing procedure. This same information can be displayed as a confidence density and a confidence curve.

The figure below depicts a meta-analysis using confidence distributions for a binomial proportion $$\theta$$. Density (a) represents an informative prior distribution based on historical data and a vague conjugate prior with an estimate of 0.90 and a sample size of n = 50. This same information is depicted in (b) as a confidence density resulting from a likelihood ratio test. A similar confidence density can be produced by inverting a Wald test with a logit link. The posterior based on the current data binomial likelihood and a vague conjugate prior is shown in (c) with an estimate of 0.87 resulting from n = 30. This same information can be represented as a likelihood ratio confidence density, (d). Using Bayes theorem, (a) and (c) combine to form (e). Multipliying the historical and current likelihoods and inverting a likelihood ratio test forms (f). This multiplication of independent likelihoods is precisely what Bayes theorem accomplishes (plus normalization), without the inversion of a hypothesis test. The confidence densities above can be interpreted under a Neyman-Pearson framework using a pre-specified null hypothesis and type I error rate. they can also be interpreted under a Fisherian framework of evidential p-values to compare the plausibility of multiple hypotheses, not necessarily pre-specified. Many Bayesians will use a Dutch book argument in support of posterior probability. Here is an example where the long-run characteristics of a likelihood ratio test are used to form a Dutch book against two different Bayesian posteriors. (3)

• I don't really get this criticism that Bayesian methods are "unfalsifiable". The classic frequentist approach is to calculate a p-value and then somehow use that p-value to make a decision, such as whether to proceed to mass manufacture of a vaccine. The frequentist may base this decision on whether the p-value is below some arbitrary threshold such as 0.05. What is falsifiable about the choice of 0.05 here? Of course, you could simply never make any decisions based on your p-values, but that raises the question of why you are even doing statistics in the first place. Aug 9, 2021 at 8:31
• Bayesian methods are unfalsifiable as long as Bayesian probabilities are taken to model belief rather than the data generating process in reality, although a Bayesian belief can turn out to be badly calibrated to reality and Bayesians can check that. However, Bayesian statistics can also be applied in a falsifiable way, see Andrew Gelman's work and in particular Sec. 5.5 here (as linked in another comment already): rss.onlinelibrary.wiley.com/doi/full/10.1111/rssa.12276 Aug 9, 2021 at 9:30
• @Geoffrey, thanks for sending the paper I will have a look at it
– giac
Aug 9, 2021 at 10:08
• Only if there exists a real-life mechanism by which we can sample values of $\theta$ can a probability distribution for $\theta$ be verified. In such settings probability statements about $\theta$ will have a purely frequentist interpretation. The challenge then is to apply the posterior probabilitly statements to the $\theta$ under investigation in a meaningful way. Aug 9, 2021 at 12:21
• @fblundun I suggest you look up what it means for something to be falsifiable. An error rate is set by design. Someone else could try to falsify this error rate by claming the pre-specified error rate is something different, but one would simply verify the error rate specified in the analysis plan. In a similar fashion a Bayesian can pre-specify a prior by design, and in that sense it is unfalsifiable, but posterior belief is not a verifiable statement about the actual parameter, the hypothesis, nor the experiment. In that sense Bayesian belief is unfalsifiable. Aug 9, 2021 at 12:30