Bayesian vs frequentist Interpretations of Probability

Question

Can someone give a good rundown of the differences between the Bayesian and the frequentist approach to probability?

From what I understand:

The frequentists view is that the data is a repeatable random sample (random variable) with a specific frequency/probability (which is defined as the relative frequency of an event as the number of trials approaches infinity). The underlying parameters and probabilities remain constant during this repeatable process and that the variation is due to variability in $X_n$ and not the probability distribution (which is fixed for a certain event/process).

The bayesian view is that the data is fixed while the frequency/probability for a certain event can change meaning that the parameters of the distribution changes. In effect, the data that you get changes the prior distribution of a parameter which gets updated for each set of data.

To me it seems that the frequentist approach is more practical/logical since it seems reasonable that events have a specific probability and that the variation is in our sampling.

Furthermore, most data analysis from studies is usually done using the frequentist approach (i.e. confidence intervals, hypothesis testing with p-values etc) since it is easily understandable.

I was just wondering whether anyone could give me a quick summary of their interpretation of bayesian vs frequentist approach including bayesian statistical equivalents of the frequentist p-value and confidence interval. In addition, specific examples of where 1 method would be preferable to the other is appreciated.

Answer 1

In the frequentist approach, it is asserted that the only sense in which probabilities have meaning is as the limiting value of the number of successes in a sequence of trials, i.e. as

$$p = \lim_{n\to\infty} \frac{k}{n}$$

where $k$ is the number of successes and $n$ is the number of trials. In particular, it doesn't make any sense to associate a probability distribution with a parameter.

For example, consider samples $X_1, \dots, X_n$ from the Bernoulli distribution with parameter $p$ (i.e. they have value 1 with probability $p$ and 0 with probability $1-p$). We can define the sample success rate to be

$$\hat{p} = \frac{X_1+\cdots +X_n}{n}$$

and talk about the distribution of $\hat{p}$ conditional on the value of $p$, but it doesn't make sense to invert the question and start talking about the probability distribution of $p$ conditional on the observed value of $\hat{p}$. In particular, this means that when we compute a confidence interval, we interpret the ends of the confidence interval as random variables, and we talk about "the probability that the interval includes the true parameter", rather than "the probability that the parameter is inside the confidence interval".

In the Bayesian approach, we interpret probability distributions as quantifying our uncertainty about the world. In particular, this means that we can now meaningfully talk about probability distributions of parameters, since even though the parameter is fixed, our knowledge of its true value may be limited. In the example above, we can invert the probability distribution $f(\hat{p}\mid p)$ using Bayes' law, to give

$$\overbrace{f(p\mid \hat{p})}^\text{posterior} = \underbrace{\frac{f(\hat{p}\mid p)}{f(\hat{p})}}_\text{likelihood ratio} \overbrace{f(p)}^\text{prior}$$

The snag is that we have to introduce the prior distribution into our analysis - this reflects our belief about the value of $p$ before seeing the actual values of the $X_i$. The role of the prior is often criticised in the frequentist approach, as it is argued that it introduces subjectivity into the otherwise austere and object world of probability.

In the Bayesian approach one no longer talks of confidence intervals, but instead of credible intervals, which have a more natural interpretation - given a 95% credible interval, we can assign a 95% probability that the parameter is inside the interval.

Answer 2

You're right about your interpretation of Frequentist probability: randomness in this setup is merely due to incomplete sampling. From the Bayesian viewpoint probabilities are "subjective", in that they reflect an agent's uncertainty about the world. It's not quite right to say that the parameters of the distributions "change". Since we don't have complete information about the parameters, our uncertainty about them changes as we gather more information.

Both interpretations are useful in applications, and which is more useful depends on the situation. You might check out Andrew Gelman's blog for ideas about Bayesian applications. In many situations what Bayesians call "priors" Frequentists call "regularization", and so (from my perspective) the excitement can leave the room rather quickly. In fact, according to the Bernstein-von Mises theorem, Bayesian and Frequentist inference are actually asymptotically equivalent under rather weak assumptions (though notably the theorem fails for infinite-dimensional distributions). You can find a slew of references about this here.

Since you asked for interpretations: I think the Frequentist viewpoint makes great sense when modeling scientific experiments as it was designed to do. For some applications in machine learning or for modeling inductive reasoning (or learning), Bayesian probability makes more sense to me. There are many situations in which modeling an event with a fixed, "true" probability seems implausible.

For a toy example going back to Laplace, consider the probability that the sun rises tomorrow. From the Frequentist perspective, we have to posit something like infinitely-many universes to define the probability. As Bayesians, there is only one universe (or at least, there needn't be many). Our uncertainty about the sun rising is squelched by our very, very strong prior belief that it will rise again tomorrow.

Answer 3

The Bayesian interpretation of probability is a degree-of-belief interpretation.

A Bayesian may say that the probability that there was life on Mars a billion years ago is $1/2$.

A frequentist will refuse to assign a probability to that proposition. It is not something that could be said to be true in half of all cases, so one cannot assign probability $1/2$.

Answer 4

Chris gives a nice simplistic explanation that properly differentiates the two approaches to probability. But frequentist theory of probability is more than just looking at the long range proportion of successes. We also consider data sampled at random from a distribution and estimate parameters of the distribution such as the mean and variance by taking certain types of averages of the data (e.g. for the mean it is the arithmetic average of the observations. Frequentist theory associates a probability with the estimate that is called the sampling distribution.

In frequency theory we are able to show for parameters like the mean that are taken by averaging from the samples that the estimate will converge to the true parameter. The sampling distribution is used to describe how close the estimate is to the parameter for any fixed sample size n. Close is defined by a measure of accuracy (e.g. mean square error).

At Chris points out for any parameter such as the mean the Bayesian attaches a prior probability distribution on it. Then given the data Bayes' rule is used to compute a posterior distribution for the parameter. For the Bayesian all inference about the parameter is based on this posterior distribution.

Frequentists construct confidence intervals which are intervals of plausible values for the parameter. Their construction is based on the frequentist probability that if the process used to generate the interval were repeated many times for independent samples the proportion of intervals that would actually include the true value of the parameter would be at least some prespecified confidence level (e.g. 95%).

Bayesians use the a posteriori distribution for the parameter to construct credible regions. These are simply regions in the parameter space over which the posterior distibution is integrated to get a prespecified probability (e.g. 0.95). Credible regions are interpreted by Bayesians as regions that have a high (e.g. the prespecified 0.95) probability of including the true value of the parameter.

Answer 5

From a "real world" point of view, I find one major difference between a frequentist and a classical or Bayesian "solution" that applies to at least three major scenarios. The difference in selecting a methodology depends on whether you need a solution that is impacted by the population probability, or one that is impacted by the individual probability. Examples below:

1. If there is a known 5% probability that males over 40 will die in a given year and require life insurance payments, an insurance company can use the 5% POPULATION percentage to estimate its costs, but to say that each individual male over 40 only has a 5% chance of dying ... is meaningless... Because 5% have a 100% probability of dying - which is a frequentist approach. At the individual level the event either occurs (100% probability) or it does not (0% probability) However, based on this limited information, it is not possible to predict the individuals who have a 100% probability of dying, and the 5% "averaged" population probability is useless at the individual level.

2. The above argument applies equally as well to fires in buildings which is why sprinklers are required in all buildings in a population.

3. Both of the above arguments apply equally as well to information systems breeches, damage, or "hacks". The population percentages are useless so all systems must be safeguarded.

Answer 6

The following is taken from my manuscript on p-value functions - Johnson, Geoffrey S. "Decision Making in Drug Development via Inference on Power" Researchgate.net (2021).

In any quantitative field it is not enough to simply apply a set of mathematical operations. One must also provide an interpretation. The field of statistics concerns itself with a special branch of mathematics regarding probability. When interpreting probability there are primarily two competing paradigms: Bayesian and frequentist. These paradigms differ on what it means for something to be considered random and what probability itself measures. Both frequentists and Bayesians would agree that once a test statistic is observed it is fixed, there is nothing random about it. Additionally, frequentists and most Bayesians would agree that the parameter under investigation, say $\theta$, is an unknown fixed quantity and it is simply treated as random in the Bayesian paradigm as a matter of practice. The question then becomes, "How do we interpret probability statements about a fixed quantity?" Without delving into the mathematical details of how a posterior or a p-value is calculated, I explore various interpretations below and what makes them untenable.

One interpretation of a Bayesian prior is that "random'' is synonymous with "unknown'' and probability measures the experimenter's belief so that the posterior measures belief about the unknown fixed true $\theta$ given the observed data. This interpretation is untenable because belief is unfalsifiable $-$ it is not a verifiable statement about the actual parameter, the hypothesis, nor the experiment. Another interpretation is that "random'' is short for "random sampling'' and probability measures the emergent pattern of many samples so that a Bayesian prior is merely a modeling assumption regarding $\theta$, i.e. the unknown fixed true $\theta$ was randomly selected from a known collection or prevalence of $\theta$'s (prior distribution) and the observed data is used to subset this collection, forming the posterior distribution. The unknown fixed true $\theta$ is now imagined to have instead been randomly selected from the posterior. This interpretation is untenable because of the contradiction caused by claiming two sampling frames. The second sampling frame is correct only if the first sampling frame is correct, yet there can only be a single sampling frame from which we obtained the unknown fixed true $\theta$ under investigation. A third interpretation of a Bayesian prior is that "random'' is synonymous with "unrealized'' or "undetermined'' and probability measures a simultaneity of existance so that $\theta$ is not fixed and all values of $\theta$ are true simultaneously; the truth exists in a superposition depending on the data observed according to the posterior distribution (think Schrödinger's cat). This interpretation is untenable because it reverses cause and effect $-$ the population-level parameter depends on the data observed, but the observed data depended on the parameter. Ascribing any of these interpretations to the posterior allows one to make philosophical probability statements about hypotheses given the data. While the p-value is typically not interpreted in the same manner, it does show us the plausibility of a hypothesis given the data $-$ the ex-post sampling probability of the observed result or something more extreme if the hypothesis for the unknown fixed $\theta$ is true.

One might notice the similarity between a p-value and a posterior probability (or a confidence interval and a credible interval) and wonder under what circumstances is each one preferable. At its essence this is a matter of scientific objectivity. To the Bayesian, probability is axiomatic and measures the experimenter. To the frequentist, probability measures the experiment and must be verifiable. The Bayesian interpretation of probability as a measure of belief is unfalsifiable. Only if there exists a real-life mechanism by which we can sample values of $\theta$ can a probability distribution for $\theta$ be verified. In such settings probability statements about $\theta$ would have a purely frequentist interpretation. This may be a reason why frequentist inference is ubiquitous in the scientific literature. If the prior distribution is chosen in such a way that the posterior is dominated by the likelihood or is proportional to the likelihood, Bayesian belief is more objectively viewed as confidence based on frequency probability of the experiment.

In short, for those who subscribe to the frequentist interpretation of probability the p-value function summarizes all the probability statements about the experiment one can make as a function of the hypothesis for $\theta$. It is a matter of correct interpretation given the definition of probability and what constitutes a random variable. The posterior remains an incredibly useful tool and can be interpreted as an approximate p-value function.

Answer 7

“I was just wondering whether anyone could give me a quick summary of their interpretation of Bayesian vs. Frequentist approach including Bayesian statistical equivalents of the Frequentist p-value and confidence interval. In addition, specific examples of where one method would be preferable to the other are appreciated.”

At a certain level, you basically have it. However, I spent some time thinking about your question and thought I would add an answer to it.

The first thing I will do is change some language. When I speak about the Frequentist perspective, I will use the words data and parameters. When I speak about the Bayesian perspective, I will use the words observables and unobservables. Do notice that they are not the same. For example, if you are missing a data point, it is an unobservable but not a parameter in the Frequentist sense. Likewise, if it is known that the variance of a process is equal to some value $k$, then it is an observable. One of the difficulties of talking across paradigms is that people try and use the same words for almost identical concepts but are not quite the same.

In the general sense, you are correct. In the Frequentist view of the world, there are fixed but unknown parameters. Those fixed parameters determine what data could be seen. The data that could be seen is the sample space. I would note that when I say “fixed,” I do not necessarily mean that $\sigma^2=k$, it could be that $\sigma^2=k*t$ or any other function or possibly relation. I do not require homoskedasticity or that a distribution is stationary. I am saying that at any instantaneous moment, there is a parameter of fixed value. Those parameters are the determiners of that which is instantaneously possible.

Note that the long-run frequencies do not follow from the observations or their nature but the fixed nature of the natural system that the frequencies represent. An essential and exceedingly helpful side effect is that the sampling distributions of many estimators can be known. In the real world, we are often dealing with estimators rather than raw frequencies. Indeed, the raw frequencies are often of little use.

It is a bit more challenging to discuss the Bayesian view of the world because it has multiple axiomatizations. In contrast, there is only one axiomatization of the Frequentist side, Kolmogorov’s. Usually, the differences do not matter, especially in applied work. Nonetheless, they can matter in theoretical work. Savage’s and de Finetti’s solutions to probability differ in some theoretical constructions, as do others. That can result in differing applied Bayesian models, particularly in the social sciences.

In addition to axiomatic differences, there can be differences in interpretation created by the silences in the math. Bayesian theory often does not require you to adopt a particular point of view, but we are humans, and we like to feel comfortable. I suspect people that work in quantum mechanics have the same difficulty.

As an example, there are two equally valid ways that you could view a prior distribution for an unobservable. The first would be that the unobserved quantity is fixed, but its location is unknown. The prior represents your uncertainty about its location. It is a representation of your beliefs about it. The second would be that Nature draws the actual value of the unobservable when you do an experiment from a distribution called the prior. Your prior represents your best estimate of what nature is doing. They are equivalent; to reject one is to deny the other. You can emotionally reject one, but not the math. You can assert one of the two, but that is a statement of emotional comfort, not math. The math doesn’t make nature do anything.

Randomness on the Bayesian side of the coin is uncertainty. There are unobservable things that you want to know more about, or to take actions about or because of, likely based on observable things. You are uncertain about the unobservables. You are certain of the observables. Please note that just because you are certain, treating the observables as fixed in the same sense as Frequentist parameters, does not imply that your observations are valid. If you observe a magician providing you with data, the observations are fixed. It does not mean that they are accurately informative of an underlying phenomenon.

Regardless, Bayesian probability measures and statistics are subjective. They depend entirely upon the prior knowledge of the system. A seasoned engineer with twenty years of experience and a graduate degree will have a different prior regarding soil samples for constructing a bridge than a fresh-out engineer who graduated engineering school last month. That difference in skill and knowledge can have very real-world consequences. A new engineer happily accepting the results of a t-test may find a grumpy senior engineer requiring more sampling and a rejection of the results by the inclusion of his or her prior.

Bayesian methods are about updating beliefs. The probability distributions are the distributions of belief. That may imply that there would be no scientific use, but that is not true. If one were to adopt the prior of an ardent opponent and show that even a highly bigoted opponent should assume the opposite belief, then that is very convincing. A passionate proponent of using Ivermectin to treat COVID, as long as they do not have a degenerate prior conviction that there is only one answer, will give up on Ivermectin as data comes through. It may take much longer than for a person with no personal opinion one way or the other, but it will happen. To be honest, because doctors prescribed so much Ivermectin in the last few months to keep their patients from going to other doctors, there is now an extensive data set. We have data from controlled experiments and natural experiments. The upshot of this is that people should get vaccinated and seek other treatments such as the monoclonal or polyclonal antibody infusions early in the infection.

As long as your prior beliefs are not degenerate, they can change upon seeing data, then the data will drive you to reality. Subject to you not living in the Truman show or gathering your entire worldview from magicians and con artists, the data wins eventually.

As to Bayesian equivalents to the p-value or the confidence interval, there are none. A p-value provides the probability of observing a result as extreme or more extreme if the null hypothesis is true. There can be no real Bayesian equivalent because there is no equivalent to the null hypothesis on the Bayesian side. No hypothesis is special, and there is no restriction to a null and an alternative. You can have any finite number of hypotheses that you find meaningful.

The closes thing is the Bayesian posterior probability. It is a statement of how much weight you give to the truth of a hypothesis. It has nothing to do with chance. The hypothesis is not assumed to be true. The question the posterior resolves is what probability do you give or how much credence or credibility do you provide a hypothesis.

There is no Bayesian equivalent to a confidence interval. A confidence interval is any function that guarantees that the interval will cover a parameter to some specified percentage of the time. There is an infinite number of such functions. Confidence intervals are not unique. Suppose you repeat an experiment an infinite number of times. In that case, the percentage of time that your interval will cover the parameter will never be less than your desired guaranteed percentage.

Of course, since infinite repetition isn’t feasible, it is just a model, as are Bayesian models.

Of great importance, if you perform an experiment and a parameter is estimated with 95% confidence to be in the interval $[a,b]$, that does not imply that there is a 95% chance that the parameter is in the interval. It is a statement of confidence in the interval building process. You believe that at least 95% of your intervals will cover the parameter as the number of experiments goes to infinity.

The closest Bayesian equivalent is the credible set. It is not an interval, and it does not have to be a connected set. It can certainly be true that the Frequentist confidence interval is $[5,15]$, when the Bayesian set of equivalent probability is $[6,7]\cup[8,9.5]$. The set can be disjoint if some area is improbable. As there is an infinite number of ways to subset a probability distribution, there is an endless number of possible credible sets that all add to at least some chosen percentage.

The credible interval is created by applying some rule to the posterior probability distribution. So a 95% credible set is the region where you would give at least a 95% probability of finding the parameter given your observations and prior knowledge.

Which method you should use depends entirely upon what usage you are going to put the techniques to.

Fisher’s method of maximum likelihood should be used whenever you want to acquire new knowledge. If you wonder if something is true, then you collect data and research it. Take that data, plug it into the method of maximum likelihood, and use that to generate a p-value or a likelihood ratio. If the p-value is small enough, then provisionally accept that your null hypothesis is false and do more research into the topic. If it is not small enough for you, then realize that you have wasted your time and go on to other, hopefully, more fruitful, things.

Pearson and Neyman’s frequency method should be used when it matters whether you accept or reject something. It allows you to create an acceptance and rejection region and gives you a way to control for statistical power. An excellent example of that would be quality control inspection. The method says that if you choose some value, $\alpha$, and stick to it, then you will be made a fool of no more often than $\alpha$ percentage of the time.

Laplace’s method of inverse probability, Bayesian analysis, should be used when you need to find something, gamble, take personal action, or update your beliefs. You should never ever place money at risk other than with Bayesian methods unless you want someone to have the ability to force you to take sure losses. Risk-taking with money is my area, so it colors my perspective. Likewise, if you need to find a downed plane, use a Bayesian method. If you need to find an unobservable quantity with observable data, Bayes is your tool. If you need to be able to make factual statements about a parameter using conventions that we can all agree with, Frequentist or Likelihoodhist methods are your toolkit.

That boils it down. The frequency side answers, “what are the minimal statements we can all agree with because we can have statistical confidence in the procedures that were used to create it?” The Bayesian side answers, “what should I believe, or how should I act, based only on what I saw and prior observations and the prior observations of other people that I have chosen to endorse?”

Answer 8

The choice of interpretation depends on the question. If you wish to know the odds in a game of chance, classical interpretation will solve your problem, but statistical data is useless since fair dice have no memory.

If you wish to predict a future event based on past experience, the frequentist interpretation is correct and sufficient.

If you don't know if a past event had occurred, and wish to assess the probability that it did, you must take your prior beliefs, i.e. what you already know about the chance of the event to occur and update your belief when you acquire new data.

Since the question is about a degree of belief, and each person may have a different idea about the priors, the interpretation is necessarily subjective, a.k.a. Bayesian.