Bayesian vs frequentist Interpretations of Probability

Question

Can someone give a good rundown of the differences between the Bayesian and the frequentist approach to probability?

From what I understand:

The frequentists view is that the data is a repeatable random sample (random variable) with a specific frequency/probability (which is defined as the relative frequency of an event as the number of trials approaches infinity). The underlying parameters and probabilities remain constant during this repeatable process and that the variation is due to variability in $X_n$ and not the probability distribution (which is fixed for a certain event/process).

The bayesian view is that the data is fixed while the frequency/probability for a certain event can change meaning that the parameters of the distribution changes. In effect, the data that you get changes the prior distribution of a parameter which gets updated for each set of data.

To me it seems that the frequentist approach is more practical/logical since it seems reasonable that events have a specific probability and that the variation is in our sampling.

Furthermore, most data analysis from studies is usually done using the frequentist approach (i.e. confidence intervals, hypothesis testing with p-values etc) since it is easily understandable.

I was just wondering whether anyone could give me a quick summary of their interpretation of bayesian vs frequentist approach including bayesian statistical equivalents of the frequentist p-value and confidence interval. In addition, specific examples of where 1 method would be preferable to the other is appreciated.

Answer 1

In the frequentist approach, it is asserted that the only sense in which probabilities have meaning is as the limiting value of the number of successes in a sequence of trials, i.e. as

$$p = \lim_{n\to\infty} \frac{k}{n}$$

where $k$ is the number of successes and $n$ is the number of trials. In particular, it doesn't make any sense to associate a probability distribution with a parameter.

For example, consider samples $X_1, \dots, X_n$ from the Bernoulli distribution with parameter $p$ (i.e. they have value 1 with probability $p$ and 0 with probability $1-p$). We can define the sample success rate to be

$$\hat{p} = \frac{X_1+\cdots +X_n}{n}$$

and talk about the distribution of $\hat{p}$ conditional on the value of $p$, but it doesn't make sense to invert the question and start talking about the probability distribution of $p$ conditional on the observed value of $\hat{p}$. In particular, this means that when we compute a confidence interval, we interpret the ends of the confidence interval as random variables, and we talk about "the probability that the interval includes the true parameter", rather than "the probability that the parameter is inside the confidence interval".

In the Bayesian approach, we interpret probability distributions as quantifying our uncertainty about the world. In particular, this means that we can now meaningfully talk about probability distributions of parameters, since even though the parameter is fixed, our knowledge of its true value may be limited. In the example above, we can invert the probability distribution $f(\hat{p}\mid p)$ using Bayes' law, to give

$$\overbrace{f(p\mid \hat{p})}^\text{posterior} = \underbrace{\frac{f(\hat{p}\mid p)}{f(\hat{p})}}_\text{likelihood ratio} \overbrace{f(p)}^\text{prior}$$

The snag is that we have to introduce the prior distribution into our analysis - this reflects our belief about the value of $p$ before seeing the actual values of the $X_i$. The role of the prior is often criticised in the frequentist approach, as it is argued that it introduces subjectivity into the otherwise austere and object world of probability.

In the Bayesian approach one no longer talks of confidence intervals, but instead of credible intervals, which have a more natural interpretation - given a 95% credible interval, we can assign a 95% probability that the parameter is inside the interval.

Answer 2

You're right about your interpretation of Frequentist probability: randomness in this setup is merely due to incomplete sampling. From the Bayesian viewpoint probabilities are "subjective", in that they reflect an agent's uncertainty about the world. It's not quite right to say that the parameters of the distributions "change". Since we don't have complete information about the parameters, our uncertainty about them changes as we gather more information.

Both interpretations are useful in applications, and which is more useful depends on the situation. You might check out Andrew Gelman's blog for ideas about Bayesian applications. In many situations what Bayesians call "priors" Frequentists call "regularization", and so (from my perspective) the excitement can leave the room rather quickly. In fact, according to the Bernstein-von Mises theorem, Bayesian and Frequentist inference are actually asymptotically equivalent under rather weak assumptions (though notably the theorem fails for infinite-dimensional distributions). You can find a slew of references about this here.

Since you asked for interpretations: I think the Frequentist viewpoint makes great sense when modeling scientific experiments as it was designed to do. For some applications in machine learning or for modeling inductive reasoning (or learning), Bayesian probability makes more sense to me. There are many situations in which modeling an event with a fixed, "true" probability seems implausible.

For a toy example going back to Laplace, consider the probability that the sun rises tomorrow. From the Frequentist perspective, we have to posit something like infinitely-many universes to define the probability. As Bayesians, there is only one universe (or at least, there needn't be many). Our uncertainty about the sun rising is squelched by our very, very strong prior belief that it will rise again tomorrow.

Answer 3

The Bayesian interpretation of probability is a degree-of-belief interpretation.

A Bayesian may say that the probability that there was life on Mars a billion years ago is $1/2$.

A frequentist will refuse to assign a probability to that proposition. It is not something that could be said to be true in half of all cases, so one cannot assign probability $1/2$.

Answer 4

Chris gives a nice simplistic explanation that properly differentiates the two approaches to probability. But frequentist theory of probability is more than just looking at the long range proportion of successes. We also consider data sampled at random from a distribution and estimate parameters of the distribution such as the mean and variance by taking certain types of averages of the data (e.g. for the mean it is the arithmetic average of the observations. Frequentist theory associates a probability with the estimate that is called the sampling distribution.

In frequency theory we are able to show for parameters like the mean that are taken by averaging from the samples that the estimate will converge to the true parameter. The sampling distribution is used to describe how close the estimate is to the parameter for any fixed sample size n. Close is defined by a measure of accuracy (e.g. mean square error).

At Chris points out for any parameter such as the mean the Bayesian attaches a prior probability distribution on it. Then given the data Bayes' rule is used to compute a posterior distribution for the parameter. For the Bayesian all inference about the parameter is based on this posterior distribution.

Frequentists construct confidence intervals which are intervals of plausible values for the parameter. Their construction is based on the frequentist probability that if the process used to generate the interval were repeated many times for independent samples the proportion of intervals that would actually include the true value of the parameter would be at least some prespecified confidence level (e.g. 95%).

Bayesians use the a posteriori distribution for the parameter to construct credible regions. These are simply regions in the parameter space over which the posterior distibution is integrated to get a prespecified probability (e.g. 0.95). Credible regions are interpreted by Bayesians as regions that have a high (e.g. the prespecified 0.95) probability of including the true value of the parameter.

Answer 5

From a "real world" point of view, I find one major difference between a frequentist and a classical or Bayesian "solution" that applies to at least three major scenarios. The difference in selecting a methodology depends on whether you need a solution that is impacted by the population probability, or one that is impacted by the individual probability. Examples below:

1. If there is a known 5% probability that males over 40 will die in a given year and require life insurance payments, an insurance company can use the 5% POPULATION percentage to estimate its costs, but to say that each individual male over 40 only has a 5% chance of dying ... is meaningless... Because 5% have a 100% probability of dying - which is a frequentist approach. At the individual level the event either occurs (100% probability) or it does not (0% probability) However, based on this limited information, it is not possible to predict the individuals who have a 100% probability of dying, and the 5% "averaged" population probability is useless at the individual level.

2. The above argument applies equally as well to fires in buildings which is why sprinklers are required in all buildings in a population.

3. Both of the above arguments apply equally as well to information systems breeches, damage, or "hacks". The population percentages are useless so all systems must be safeguarded.

Answer 6

The following is taken from my manuscript on confidence distributions - Johnson, Geoffrey S. "Decision Making in Drug Development via Confidence Distributions." arXiv preprint arXiv:2005.04721 (2020).

In the Bayesian framework the population-level parameter of interest is considered an unrealized or unobservable realization of a random variable that depends on the observed data.  This premise has cause and effect reversed.  In order to overcome this the Bayesian approach reinterprets probability as measuring the subjective belief of the experimenter. This is one interpretation of the Bayesian posterior distribution.  Another interpretation is that the unknown fixed parameter, say theta, was randomly selected from a known collection or prevalence of theta's (prior distribution) and the observed data is used to subset this collection, forming the posterior.  The unknown fixed true theta is now imagined to have instead been randomly selected from the posterior.  Every time the prior or posterior is updated the sampling frame from where we obtained our unknown fixed true theta under investigation must be changed.  A third interpretation is that all values of theta are true simultaneously.  The truth exists in a superposition depending on the evidence observed (think Schrodinger's cat).  Ascribing any of these interpretations to the posterior allows one to make philosophical probability statements about hypotheses given the data.  While the p-value is typically not interpreted in the same manner, it does show us the plausibility of a hypothesis given the data - the ex-post sampling probability of the observed result or something more extreme if the hypothesis is true.  This does not reverse cause and effect.

One might notice the similarity between the p-value and a posterior probability and wonder under what circumstances is each one preferable.  At its essence this is a matter of scientific objectivity.  To the Bayesian, probability is axiomatic and measures the experimenter.  To the frequentist, probability measures the experiment and must be verifiable.  The Bayesian interpretation of probability as a measure of belief is unfalsifiable. Only if there exists a real-life mechanism by which we can sample values of theta can a probability distribution for theta be verified.  In such settings probability statements about theta would have a purely frequentist interpretation (see the second interpretation of the posterior above).  This may be a reason why frequentist inference is ubiquitous in the scientific literature.

The interpretation of frequentist inference is straight forward for non-statisticians by citing confidence levels, e.g. 'We are 15.9% confident that theta is less than or equal to theta_0.'  Of course to fully appreciate this statement of confidence one must more fully define the p-value as a frequency probability of the experiment if the hypothesis is true (think of the proof by contradiction structure a prosecutor uses in a court room setting, innocent until proven guilty).  A Bayesian approach may make it easy for some to interpret a posterior probability, e.g. 'There is 17.4% Bayesian belief probability that theta is less than or equal to theta_0.'  Of course to fully appreciate this statement one must fully define Bayesian belief and make it clear this is not a verifiable statement about the actual parameter, the hypothesis, nor the experiment.  If the prior distribution is chosen in such a way that the posterior is dominated by the likelihood or is proportional to the likelihood, Bayesian belief is more objectively viewed as confidence based on frequency probability of the experiment.  In short, for those who subscribe to the frequentist interpretation of probability, the confidence distribution summarizes all the probability statements about the experiment one can make.  It is a matter of correct interpretation given the definition of probability and what constitutes a random variable.  The posterior remains an incredibly useful tool and can be interpreted as a valid asymptotic confidence distribution. Historical data can be incorporated into a frequentist approach through a fixed-effect meta-analysis.

Answer 7

The choice of interpretation depends on the question. If you wish to know the odds in a game of chance, classical interpretation will solve your problem, but statistical data is useless since fair dice have no memory.

If you wish to predict a future event based on past experience, the frequentist interpretation is correct and sufficient.

If you don't know if a past event had occurred, and wish to assess the probability that it did, you must take your prior beliefs, i.e. what you already know about the chance of the event to occur and update your belief when you acquire new data.

Since the question is about a degree of belief, and each person may have a different idea about the priors, the interpretation is necessarily subjective, a.k.a. Bayesian.