37
$\begingroup$

Can someone give a good rundown of the differences between the Bayesian and the frequentist approach to probability?

From what I understand:

The frequentists view is that the data is a repeatable random sample (random variable) with a specific frequency/probability (which is defined as the relative frequency of an event as the number of trials approaches infinity). The underlying parameters and probabilities remain constant during this repeatable process and that the variation is due to variability in $X_n$ and not the probability distribution (which is fixed for a certain event/process).

The bayesian view is that the data is fixed while the frequency/probability for a certain event can change meaning that the parameters of the distribution changes. In effect, the data that you get changes the prior distribution of a parameter which gets updated for each set of data.

To me it seems that the frequentist approach is more practical/logical since it seems reasonable that events have a specific probability and that the variation is in our sampling.

Furthermore, most data analysis from studies is usually done using the frequentist approach (i.e. confidence intervals, hypothesis testing with p-values etc) since it is easily understandable.

I was just wondering whether anyone could give me a quick summary of their interpretation of bayesian vs frequentist approach including bayesian statistical equivalents of the frequentist p-value and confidence interval. In addition, specific examples of where 1 method would be preferable to the other is appreciated.

|cite|improve this question
$\endgroup$
  • 1
    $\begingroup$ In some venues you'll be attacked by an angry mob if you say the frequentist approach to statistical inference is more practical. (OK, maybe there's some hyperbole in that statement.) I don't agree that confidence intervals are easier to understand than posterior probability intervals. (Anyway, see my answer below. I think it gets straight to the essence of the matter, although there's no math beyond knowing what $1/2$ is.) $\endgroup$ – Michael Hardy Jul 5 '12 at 19:51
  • $\begingroup$ @DilipSarwate ay, I'll keep that in mind for next time. but seems like I got a few good answers this time so maybe I'll try to finish off here :D $\endgroup$ – BYS2 Jul 6 '12 at 15:23
  • $\begingroup$ See also stats.stackexchange.com/q/173056/35989 $\endgroup$ – Tim♦ Oct 20 '18 at 9:10
27
$\begingroup$

In the frequentist approach, it is asserted that the only sense in which probabilities have meaning is as the limiting value of the number of successes in a sequence of trials, i.e. as

$$p = \lim_{n\to\infty} \frac{k}{n}$$

where $k$ is the number of successes and $n$ is the number of trials. In particular, it doesn't make any sense to associate a probability distribution with a parameter.

For example, consider samples $X_1, \dots, X_n$ from the Bernoulli distribution with parameter $p$ (i.e. they have value 1 with probability $p$ and 0 with probability $1-p$). We can define the sample success rate to be

$$\hat{p} = \frac{X_1+\cdots +X_n}{n}$$

and talk about the distribution of $\hat{p}$ conditional on the value of $p$, but it doesn't make sense to invert the question and start talking about the probability distribution of $p$ conditional on the observed value of $\hat{p}$. In particular, this means that when we compute a confidence interval, we interpret the ends of the confidence interval as random variables, and we talk about "the probability that the interval includes the true parameter", rather than "the probability that the parameter is inside the confidence interval".

In the Bayesian approach, we interpret probability distributions as quantifying our uncertainty about the world. In particular, this means that we can now meaningfully talk about probability distributions of parameters, since even though the parameter is fixed, our knowledge of its true value may be limited. In the example above, we can invert the probability distribution $f(\hat{p}\mid p)$ using Bayes' law, to give

$$\overbrace{f(p\mid \hat{p})}^\text{posterior} = \underbrace{\frac{f(\hat{p}\mid p)}{f(\hat{p})}}_\text{likelihood ratio} \overbrace{f(p)}^\text{prior}$$

The snag is that we have to introduce the prior distribution into our analysis - this reflects our belief about the value of $p$ before seeing the actual values of the $X_i$. The role of the prior is often criticised in the frequentist approach, as it is argued that it introduces subjectivity into the otherwise austere and object world of probability.

In the Bayesian approach one no longer talks of confidence intervals, but instead of credible intervals, which have a more natural interpretation - given a 95% credible interval, we can assign a 95% probability that the parameter is inside the interval.

|cite|improve this answer
$\endgroup$
  • 6
    $\begingroup$ On the other hand, one criticism of the frequentist approach is that it doesn't square with how people think about probability. Consider how people talk about the "probability" of one-off events like the extinction of the dinosaurs, or the "probability" of "certainties" like the sun rising tomorrow... $\endgroup$ – Zhen Lin Jul 5 '12 at 16:08
  • 14
    $\begingroup$ It might also be good to mention that the gap between the frequentist and Bayesian approaches is not nearly as great on a practical level: any frequentist method that produces useful and self-consistent results can generally be given a Bayesian interpretation, and vice versa. In particular, recasting a frequentist calculation in Bayesian terms typically yields a rule for calculating the posterior given some specific prior. One can then ask "Well, is that prior actually a reasonable one to assume?" $\endgroup$ – Ilmari Karonen Jul 5 '12 at 16:22
  • $\begingroup$ Thank you for this answer, it is in line with my general understanding. However, I was wondering if you could clarify one thing, how would you find the probability of the data/sample success rate (f (p-hat)) in Baye's law formula? I have read through some worked examples and I generally understand how to derive f(p-hat | p) and the prior f(p) but f(p-hat) eludes me so far. If you had some links to some resources then that would be terrific :D. Thanks! $\endgroup$ – BYS2 Jul 6 '12 at 15:19
  • $\begingroup$ @IlmariKaronen. Ok so are you saying that say if I had a study which produced certain results expressed as confidence intervals, I could recast the data and do a bayesian analysis instead? and the results would be more or less consistent? $\endgroup$ – BYS2 Jul 6 '12 at 15:21
  • $\begingroup$ What @Karonen says is not entirely accurate. The two most common frequentist techniques are point estimates (usually maximum likelihood estimation) and hypothesis tests, and neither can really be given a natural Bayesian interpretation. $\endgroup$ – Jules Dec 19 '15 at 0:08
20
$\begingroup$

You're right about your interpretation of Frequentist probability: randomness in this setup is merely due to incomplete sampling. From the Bayesian viewpoint probabilities are "subjective", in that they reflect an agent's uncertainty about the world. It's not quite right to say that the parameters of the distributions "change". Since we don't have complete information about the parameters, our uncertainty about them changes as we gather more information.

Both interpretations are useful in applications, and which is more useful depends on the situation. You might check out Andrew Gelman's blog for ideas about Bayesian applications. In many situations what Bayesians call "priors" Frequentists call "regularization", and so (from my perspective) the excitement can leave the room rather quickly. In fact, according to the Bernstein-von Mises theorem, Bayesian and Frequentist inference are actually asymptotically equivalent under rather weak assumptions (though notably the theorem fails for infinite-dimensional distributions). You can find a slew of references about this here.

Since you asked for interpretations: I think the Frequentist viewpoint makes great sense when modeling scientific experiments as it was designed to do. For some applications in machine learning or for modeling inductive reasoning (or learning), Bayesian probability makes more sense to me. There are many situations in which modeling an event with a fixed, "true" probability seems implausible.

For a toy example going back to Laplace, consider the probability that the sun rises tomorrow. From the Frequentist perspective, we have to posit something like infinitely-many universes to define the probability. As Bayesians, there is only one universe (or at least, there needn't be many). Our uncertainty about the sun rising is squelched by our very, very strong prior belief that it will rise again tomorrow.

|cite|improve this answer
$\endgroup$
17
$\begingroup$

The Bayesian interpretation of probability is a degree-of-belief interpretation.

A Bayesian may say that the probability that there was life on Mars a billion years ago is $1/2$.

A frequentist will refuse to assign a probability to that proposition. It is not something that could be said to be true in half of all cases, so one cannot assign probability $1/2$.

|cite|improve this answer
$\endgroup$
  • 2
    $\begingroup$ There is probably no better place to ponder the limitations of the more narrow frequentist approach vs. the generality of the Bayesian approach (extension of logic) than the classic paper by R. T. Cox. $\endgroup$ – gwr Jul 5 '18 at 15:23
  • 2
    $\begingroup$ Cox also wrote a book about this, titled Algebra of Probable Inference, published by Johns Hopkins. @gwr $\qquad$ $\endgroup$ – Michael Hardy Jul 6 '18 at 18:03
  • 1
    $\begingroup$ Ian Hacking said it well in his book "An Introduction to Probability and Inductive Logic". He said: "The Bayesian is able to attach personal probabilities, or degrees of belief, to individual propositions. The hard-line frequency dogmatist thinks that probabilities can be attached only to a series of events." $\endgroup$ – Buttons840 Dec 23 '18 at 5:31
9
$\begingroup$

Chris gives a nice simplistic explanation that properly differentiates the two approaches to probability. But frequentist theory of probability is more than just looking at the long range proportion of successes. We also consider data sampled at random from a distribution and estimate parameters of the distribution such as the mean and variance by taking certain types of averages of the data (e.g. for the mean it is the arithmetic average of the observations. Frequentist theory associates a probability with the estimate that is called the sampling distribution.

In frequency theory we are able to show for parameters like the mean that are taken by averaging from the samples that the estimate will converge to the true parameter. The sampling distribution is used to describe how close the estimate is to the parameter for any fixed sample size n. Close is defined by a measure of accuracy (e.g. mean square error).

At Chris points out for any parameter such as the mean the Bayesian attaches a prior probability distribution on it. Then given the data Bayes' rule is used to compute a posterior distribution for the parameter. For the Bayesian all inference about the parameter is based on this posterior distribution.

Frequentists construct confidence intervals which are intervals of plausible values for the parameter. Their construction is based on the frequentist probability that if the process used to generate the interval were repeated many times for independent samples the proportion of intervals that would actually include the true value of the parameter would be at least some prespecified confidence level (e.g. 95%).

Bayesians use the a posteriori distribution for the parameter to construct credible regions. These are simply regions in the parameter space over which the posterior distibution is integrated to get a prespecified probability (e.g. 0.95). Credible regions are interpreted by Bayesians as regions that have a high (e.g. the prespecified 0.95) probability of including the true value of the parameter.

|cite|improve this answer
$\endgroup$
  • 1
    $\begingroup$ Credible regions are interpreted by Bayesians as regions that have a high (e.g. the prespecified 0.95) probability of including the true value of the parameter. How is this possible if the parameter is a random variable? $\endgroup$ – user10525 Jul 5 '12 at 19:39
  • $\begingroup$ @Procrastinator Okay maybe you would prefer for me to just say that it covers a high prespecified proportion of the parameter distribution. But if X is a random variable with a distribution f and we construct a credible region for it then the region does represent the probability that a realization of the random variable will lie in the region. $\endgroup$ – Michael R. Chernick Jul 5 '12 at 21:09
  • $\begingroup$ I agree with this explanation. It is important to clarify that a realisation of the random variable is not the true value of the parameter. $\endgroup$ – user10525 Jul 6 '12 at 9:22
  • $\begingroup$ @Procrastinator that's an interesting point you raise. However, my understanding of bayesian probability is that many Bayesians agree with classical statisticians that there is a single TRUE value of the parameter in question (it is fixed but unknown). It is the uncertainty about this parameter that is distributed because of our imperfect state of knowledge. So if you think about it in this way, then Michael Chernick's initial statement is valid don't you think? $\endgroup$ – BYS2 Jul 6 '12 at 15:33
  • 2
    $\begingroup$ @MichaelChernick I think there is a missinterpretation of what a Bayesian credibility region means. Suppose that the true value of the parameter is $\theta_0=1$ and you choose a uniform prior on $(1,100)$. Therefore no credibility interval would contain the true value of the parameter, contradicting your reasoning. $\endgroup$ – user10525 Jul 7 '12 at 11:39
2
$\begingroup$

From a "real world" point of view, I find one major difference between a frequentist and a classical or Bayesian "solution" that applies to at least three major scenarios. The difference in selecting a methodology depends on whether you need a solution that is impacted by the population probability, or one that is impacted by the individual probability. Examples below:

  1. If there is a known 5% probability that males over 40 will die in a given year and require life insurance payments, an insurance company can use the 5% POPULATION percentage to estimate its costs, but to say that each individual male over 40 only has a 5% chance of dying ... is meaningless... Because 5% have a 100% probability of dying - which is a frequentist approach. At the individual level the event either occurs (100% probability) or it does not (0% probability) However, based on this limited information, it is not possible to predict the individuals who have a 100% probability of dying, and the 5% "averaged" population probability is useless at the individual level.

  2. The above argument applies equally as well to fires in buildings which is why sprinklers are required in all buildings in a population.

  3. Both of the above arguments apply equally as well to information systems breeches, damage, or "hacks". The population percentages are useless so all systems must be safeguarded.

|cite|improve this answer
$\endgroup$
  • 2
    $\begingroup$ I do not recognize a frequentist approach in any of these three instances. They all seem to hinge on a retrospective--and therefore useless--concept of probability which is not used in classical models. For instance, the assertion that "the event either occurs ... or it does not" is trivially true but unrelated to probabilities. $\endgroup$ – whuber♦ May 25 '16 at 13:27
0
$\begingroup$

The choice of interpretation depends on the question. If you wish to know the odds in a game of chance, classical interpretation will solve your problem, but statistical data is useless since fair dice have no memory.

If you wish to predict a future event based on past experience, the frequentist interpretation is correct and sufficient.

If you don't know if a past event had occurred, and wish to assess the probability that it did, you must take your prior beliefs, i.e. what you already know about the chance of the event to occur and update your belief when you acquire new data.

Since the question is about a degree of belief, and each person may have a different idea about the priors, the interpretation is necessarily subjective, a.k.a. Bayesian.

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.