Where did the frequentist-Bayesian debate go?

Question

The world of statistics was divided between frequentists and Bayesians. These days it seems everyone does a bit of both. How can this be? If the different approaches are suitable for different problems, why did the founding fathers of statistics did not see this? Alternatively, has the debate been won by Frequentists and the true subjective Bayesians moved to decision theory?

Answer 1

I actually mildly disagree with the premise. Everyone is a Bayesian, if they really do have a probability distribution handed to them as a prior. The trouble comes about when they don't, and I think there's still a pretty good-sized divide on that topic.

Having said that, though, I do agree that more and more people are less inclined to fight holy wars and just get on with doing what seems appropriate in any given situation.

I would say that, as the profession advanced, both sides realized there were merits in the other side's approaches. Bayesians realized that evaluating how well Bayesian procedures would do if used over and over again (e.g., does this 95% credible interval (CI) actually contain the true parameter about 95% of the time?) required a frequentist outlook. Without this, there's no calibration of that "95%" to any real-world number. Robustness? Model building through iterative fitting etc.? Ideas that came up in the frequentist world, and were adapted by Bayesians starting in the late 1980s or so. Frequentists realized that regularization was good, and use it quite commonly these days - and Bayesian priors can be easily interpreted as regularization. Nonparametric modeling via cubic splines with a penalty function? Your penalty is my prior! Now we can all get along.

The other major influence, I believe, is the staggering improvement in availability of high-quality software that will let you do analysis quickly. This comes in two parts - algorithms, e.g., Gibbs sampling and Metropolis-Hastings, and the software itself, R, SAS, ... I might be more of a pure Bayesian if I had to write all my code in C (I simply wouldn't have the time to try anything else), but as it is, I'll use gam in the mgcv package in R any time my model looks like I can fit it into that framework without too much squeezing, and I'm a better statistician for it. Familiarity with your opponent's methods, and realizing how much effort it can save / better quality it can provide to use them in some situations, even though they may not fit 100% into your default framework for thinking about an issue, is a big antidote to dogmatism.

Answer 2

(The original answer is dated 2012).

This is a difficult question to answer. The number of people who truly do both is still very limited. Hard core Bayesians despise the users of mainstream statistics for their use of $p$-values, a nonsensical, internally inconsistent statistic for Bayesians; and the mainstream statisticians just do not know Bayesian methods well enough to comment on them. In the light of this, you will see a lot of criticism of the null hypothesis significance testing in Bayesian literature (ranging as far as nearly pure biology or pure psychology journals), with little to no response from mainstreamers.

There are conflicting manifestation as to "who won the debate" in statistics profession. On one hand, the the composition of an average statistics department is that in most places, you will find 10-15 mainstreamers vs. 1-2 Bayesians, although some departments are purely Bayesian, with no mainstreamers at all, except probably for consulting positions that have the job responsibility of producing experiment designs for biologists. Harvard, Duke, Carnegie Mellon, British Columbia, Montreal come to mind in North America; I am less familiar with European scene. On the other hand, you will see that in journals like JASA or JRSS, probably 25-30% of papers are Bayesian. In a way, the Bayesian renaissance may be something like the burst of ANOVA papers in the 1950s: back then, people thought that pretty much any statistics problem can be framed as an ANOVA problem; right now, people think that pretty much anything can be solved with the right MCMC.

My feeling is that applied areas don't bother figuring out the philosophical details, and just go with whatever is easier to work with. Bayesian methodology is just too damn complicated: on top of statistics, you also need to learn the art of computation (setting up the sampler, blocking, convergence diagnostics, blah-blah-blah) and be prepared to defend your priors (should you use objective priors, or should you use informative priors if the field has pretty much settled on the speed of light being 3e8 m/s, or even whether the choice of the prior affects whether your posterior will be proper or not). So in most medical or psychology or economics applications, you will see mainstream approaches in the papers written by substantive researchers, although you can also see occasional glimpse of a Bayesian paper -- to have been written by more sophisticated methodologists, or in collaboration with a Bayesian statistician, just because that was the person available at a local department to do this collaborative work.

One area where, I think, Bayesian framework is still coming short is model diagnostics -- and that is an important area for practitioners. In Bayesian world, to diagnose a model, you need to build a more complicated one and choose whichever has a better fit by Bayesian factor or BIC. So if you don't like the normality assumption for your linear regression, you can build a regression with Student errors, and let the data generate an estimate of the degrees of freedom, or you can become all fancy and have a Dirichlet process for your error terms and do some M-H jumps between different models. The mainstream approach would be to build a Q-Q plot of studentized residuals and remove outliers, and this is, again, so much simpler.

I edited a chapter in a book on this -- see http://onlinelibrary.wiley.com/doi/10.1002/9780470583333.ch5/summary. It is a very archetypal paper, in that gave about 80 references on this debate, all supporting the Bayesian point of view. (I asked the author to extend it in a revised version, which says a lot about it :) ). Jim Berger from Duke, one of the leading Bayesian theorists, gave a number of lectures, and wrote a number of very thoughtful articles on the topic.

P.S. (edit in June 2020): the "blah-blah-blah" part of computation has been significantly simplified in the recent years with Stan (https://mc-stan.org/). The NUTS sampler has fewer parameters to tweak, while offering additional diagnostics that make convergence failures more obvious. Model diagnostics has seen improvements, too, with posterior predictive checks and simulation based calibration.

Answer 3

There is a good reason for still having both, which is that a good craftsman will want to select the best tool for the task at hand, and both Bayesian and frequentist methods have applications where they are the best tool for the job.

However, often the wrong tool for the job is used because frequentist statistics are more amenable to a "statistics cookbook" approach which makes them easier to apply in science and engineering than their Bayesian counterparts, even though the Bayesian methods provide a more direct answer to the question posed (which is generally what we can infer from the particular sample of data we actually have). I am not greatly in favour of this as the "cookbook" approach leads to using statistics without a solid understanding of what you are actually doing, which is why things like the p-value fallacy crop up again and again.

However, as time progresses, the software tools for the Bayesian approach will improve and they will be used more frequently as jbowman rightly says.

I am a Bayesian by inclination (it seems to make a lot more sense to me than the frequentist approach), however I end up using frequentist statistics in my papers, partly because I will have trouble with the reviewers if I use Bayesian statistics as they will be "non-standard".

Finally (somewhat tongue in cheek ;o), to quote Max Plank "A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it."

Answer 4

I don't think the Frequentists and Bayesians give different answers to the same questions. I think they are prepared to answer different questions. Therefore, I don't think it makes sense to talk much about one side winning, or even to talk about compromise.

Consider all the questions we might want to ask. Many are just impossible questions ("What is the true value of $\theta$?"). It's more useful to consider the subset of these questions that can be answered given various assumptions. The larger subset is the questions that can be answered where you do allow yourself to use priors. Call this set BF. There is a subset of BF, which is the set of questions that do not depend on any prior. Call this second subset F. F is a subset of BF. Define B = BF \ B.

However, we cannot choose which questions to answer. In order to make useful inferences about the world, we sometimes have to answer questions that are in B and that means using a prior.

Ideally, given an estimator you would do a thorough analysis. You might use a prior, but it also would be cool if you could prove nice things about your estimator which do not depend on any prior. That doesn't mean you can ditch the prior, maybe the really interesting questions require a prior.

Everybody agrees on how to answer the questions in F. The worry is whether the really 'interesting' questions are in F or in B?

An example: a patient walks into the doctor and is either healthy(H) or sick(S). There is a test that we run, which will return positive(+) or negative(-). The test never gives false negatives - i.e $\mathcal{P}(-|S) = 0$. But it will sometimes give false positives - $\mathcal{P}(+|H) = 0.05$

We have a piece of card and the testing machine will write + or - on one side of the card. Imagine, if you will, that we have an oracle who somehow knows the truth, and this oracle writes the true state, H or S, on the other side of the card before putting the card into an envelope.

As the statistically-trained doctor, what can we say about the card in the envolope before we open the card? The following statements can be made (these are in F above):

• If S on one side of the card, then the other side will be +. $\mathcal{P}(+|S) = 1$
• If H, then the other side will be + with 5% probability, - with 95% probability. $\mathcal{P}(-|H) = 0.95$
• (summarizing the last two points) The probability that the two sides match is at least 95%. $\mathcal{P}( (-,S) \cup (+,H) ) \geq 0.95$

We don't know what $\mathcal{P}( (-,S) )$ or $\mathcal{P}( (+,H) )$ is. We can't really answer that without some sort of prior for $\mathcal{P}(S)$. But we can make statements about the sum of those two probabilities.

This is as far as we can go so far. Before opening the envelope, we can make very positive statements about the accuracy of the test. There is (at least) 95% probability that the test result matches the truth.

But what happens when we actually open the card? Given that the test result is positive (or negative), what can we say about whether they are healthy or sick?

If the test is positive (+), there is nothing we can say. Maybe they are healthy, and maybe not. Depending on the current prevalence of the disease ($\mathcal{P}(S)$) it might be the case that most patients who test positive are healthy, or it might be the case that most are sick. We can't put any bounds on this, without first allowing ourselves to put some bounds on $\mathcal{P}(S)$.

In this simple example, it's clear that everybody with a negative test result is healthy. There are no false negatives, and hence every statistician will happily send that patient home. Therefore, it makes no sense to pay for the advice of a statistician unless the test result has been positive.

The three bullet points above are correct, and quite simple. But they're also useless! The really interesting question, in this admittedly contrived model, is:

$$ \mathcal{P}(S|+) $$

and this cannot be answered without $\mathcal{P}(S)$ (i.e a prior, or at least some bounds on the prior)

I don't deny this is perhaps an oversimplified model, but it does demonstrate that if we want to make useful statements about the health of those patients, we must start off we some prior belief about their health.

Answer 5

As you'll see, there's quite a lot of frequentist-Bayesian debate going on. In fact, I think it's hotter than ever, and less dogmatic. You might be interested in my blog: http://errorstatistics.com

Answer 6

Many people (outside the specialist experts) who think they are frequentist are in fact Bayesian. This makes the debate a bit pointless. I think that Bayesianism won, but that there are still many Bayesians who think they are frequentist. There are some people who think that they don't use priors and hence they think they are frequentist. This is dangerous logic. This is not so much about priors (uniform priors or non-uniform), the real difference is more subtle.

(I'm not formally in the statistics department; my background is maths and computer science. I'm writing because of difficulties I've had trying to discuss this 'debate' with other non-statisticians, and even with some early-career statisticians.)

The MLE is actually a Bayesian method. Some people will say "I'm a frequentist because I use the MLE to estimate my parameters". I have seen this in peer-reviewed literature. This is nonsense and is based on this (unsaid, but implied) myth that a frequentist is somebody who uses a uniform prior instead of a non-uniform prior).

Consider drawing a single number from a normal distribution with known mean, $\mu = 0$, and unknown variance. Call this variance $\theta$.

$ X \equiv N(\mu = 0, \sigma^2 = \theta) $

Now consider the likelihood function. This function has two parameters, $x$ and $\theta$ and it returns the probability, given $\theta$, of $x$.

$ f(x,\theta) = \mathrm{P}_{\sigma^2=\theta} (X=x) = \frac{1}{\sqrt{2\pi \theta}} e^{-\frac{x^2}{2\theta}} $

You can imagine plotting this in a heatmap, with $x$ on the x-axis and $\theta$ on the y-axis, and using the colour (or z-axis). Here is the plot, with contour lines and colours.

First, a few observations. If you fix on a single value of $\theta$, then you can take the corresponding horizontal slice through the heatmap. This slice will give you the pdf for that value of $\theta$. Obviously, the area under the curve in that slice will be 1. On the other hand, if you fix on a single value of $x$, and then look at the corresponding vertical slice, then there is no such guarantee about the area under the curve.

This distinction between the horizontal and vertical slices is crucial, and I found this analogy helped me to understand the frequentist approach to bias.

A Bayesian is somebody who says

For this value of x, which values of $\theta$ give a 'high enough' value of $f(x,\theta)$?.

Alternatively, a Bayesian might include a prior, $g(\theta)$, but they are still talking about

for this value of x, which values of $\theta$ give a high enough value of $f(x,\theta)g(\theta)$?

So a Bayesian fixes x and looks at the corresponding vertical slice in that contour plot (or in the variant plot incorporating the prior). In this slice, the area under the curve need not be 1 (as I said earlier). A Bayesian 95% credible interval (CI) is the interval which contains 95% of the available area. For example, if the area is 2, then the area under the Bayesian CI must be 1.9.

On the other hand, a frequentist will ignore x and first consider fixing $\theta$, and will ask:

For this $\theta$, which values of x will appear most often?

In this example, with $\mathcal{N}(\mu=0, \sigma^2 = \theta)$, one answer to this frequentist question is: "For a given $\theta$, 95% of the $x$ will appear between $-3\sqrt\theta$ and $+3\sqrt\theta$."

So a frequentist is more concerned with the horizontal lines corresponding to fixed values of $\theta$.

This is not the only way to construct the frequentist CI, it's not even a good (narrow) one, but bear with me for a moment.

The best way to interpret the word 'interval' is not as an interval on a 1-d line, but to think of it as an area on the above 2-d plane. An 'interval' is a subset of the 2-d plane, not of any 1-d line. If somebody proposes such an 'interval', we then have to test is the 'interval' is valid at a 95% confidence/credible level.

A frequentist will check the validity of this 'interval' by considering each horizontal slice in turn and looking at the area under the curve. As I said before, the area under this curve will always be one. The crucial requirement is that the area within the 'interval' be at least 0.95.

A Bayesian will check validity by instead looking at the vertical slices. Again, the area under the curve will be compared to the subarea that's under the interval. If the latter is at least 95% of the former, then the 'interval' is a valid 95% Bayesian credible interval.

Now that we know how to test whether a particular interval is 'valid', the question is how do we choose the best option among the valid options. This can be a black art, but generally you want the narrowest interval. Both approaches tend to agree here - the vertical slices are considered and the goal is to make the interval as narrow as possible within each vertical slice.

I have not attempted to define the narrowest possible frequentist confidence interval in the above example. See the comments by @cardinal below for examples of narrower intervals. My goal is not to find the best intervals, but to emphasize the difference between the horizontal and vertical slices in determining validity. An interval that satisfies the conditions of a 95% frequentist confidence interval will usually not satisfy the conditions of a 95% Bayesian credible interval, and vice versa.

Both approaches desire narrow intervals, i.e. when considering one vertical slice we want to make the (1-d) interval in that slice to be as narrow as possible. The difference is in how the 95% is enforced - a frequentist will only look at proposed intervals where 95% of each horizontal slice's area is under the interval, whereas a Bayesian will insist that each vertical slice be such that 95% of its area is under the interval.

Many non-statisticians don't understand this and they focus only on the vertical slices; this makes them Bayesians even if they think otherwise.

Answer 7

These are actually apples and oranges if you dig deep enough.

The Frequentist vs. Bayesian approach dilemma is actually a philosophical question about what probabilities actually are.

Bayes discovered an useful tool to handle probabilities and treated them as primary entities in his mathematical background. Fisher, Pearson, Gosset and others discovered useful tools and framed them within the hypothetic deductive perspective. The frequentists regard probabilities as reflecting outcomes happening most often in deterministic systems, Bayesians consider probabilities to be a more fundamental concept reflecting the distribution of possible configurations in a random system.

In practical terms, we call Bayesian models those who are based on Bayes theorem, calculating posterior probabilities based on a prior, a likelihood function and model evidence. Frequentist models are slightly hard to define, because that is a label for many different statistical frameworks. Generally, it means p-values and estimation using exact maximum likelihood solutions via least squares method.

One would say Bayesian analysis gives probability distributions instead of testing hypothesis in a classical manner (e.g. null hypothesis). This is also a misunderstanding, since testing falsifiable scenarios is an epistemological choice and could be done with any tools (including non mathematical ones).