# When (if ever) is a frequentist approach substantively better than a Bayesian?

Background: I do not have an formal training in Bayesian statistics (though I am very interested in learning more), but I know enough--I think--to get the gist of why many feel as though they are preferable to Frequentist statistics. Even the undergraduates in the introductory statistics (in social sciences) class I am teaching find the Bayesian approach appealing--"Why are we interested in calculating the probability of the data, given the null? Why can't we just quantify the probability of the null hypothesis? Or the alternative hypothesis? And I've also read threads like these, which attest to the empirical benefits of Bayesian statistics as well. But then I came across this quote by Blasco (2001; emphasis added):

If the animal breeder is not interested in the philosophical problems associated with induction, but in tools to solve problems, both Bayesian and frequentist schools of inference are well established and it is not necessary to justify why one or the other school is preferred. Neither of them now has operational difficulties, with the exception of some complex cases...To choose one school or the other should be related to whether there are solutions in one school that the other does not offer, to how easily the problems are solved, and to how comfortable the scientist feels with the particular way of expression results.

The Question: The Blasco quote seems to suggest that there might be times when a Frequentist approach is actually preferable to a Bayesian one. And so I am curious: when would a frequentist approach be preferable over a Bayesian approach? I'm interested in answers that tackle the question both conceptually (i.e., when is knowing the probability of the data conditioned on the null hypothesis especially useful?) and empirically (i.e., under what conditions do Frequentist methods excel vs. Bayesian?).

It would also be preferable if answers were conveyed as accessibly as possible--it would be nice to take some responses back to my class to share with my students (though I understand some level of technicality is required).

Finally, despite being a regular user of Frequentist statistics, I am actually open to the possibility that Bayesian just wins across the board.

• When you deal with objective probabilities, i.e. naturally stochastic processes. For instance, radioactive decay has nothing to do with your subjective beliefs or unknown information, or pretty much anything else. It just goes its own pace, and atoms truly randomly break up. – Aksakal Feb 4 '16 at 16:31
• See this recent question that unfortunately ended up closed as too broad (I voted to reopen but it never was): stats.stackexchange.com/questions/192572. You are asking almost exactly the same thing. Check the answer there. – amoeba Feb 4 '16 at 16:34
• @Aksakal: I would love to have this discussion but it's off-topic and we will be told off so I shut up (and calculate). – amoeba Feb 4 '16 at 16:51
• "Bayesians address the question everyone is interested in by using assumptions no-one believes, while frequentists use impeccable logic to deal with an issue of no interest to anyone" -- Louis Lyons – Ruggero Turra Feb 4 '16 at 18:42
• @jsakaluk, notice how Bayesians' strongholds are areas where there's no enough data or when the processes are unstable, i.e. social sciences, psudo sciences, life sciences etc. There's no need to be Bayesian in quantum mechanics or most of physics. Granted, you can be Bayesian there too, it's just your inferences will be no different from frequentist's – Aksakal Feb 4 '16 at 18:46

Here's five reasons why frequentists methods may be preferred:

• Faster. Given that Bayesian statistics often give nearly identical answers to frequentist answers (and when they don't, it's not 100% clear that Bayesian is always the way to go), the fact that frequentist statistics can be obtained often several orders of magnitude faster is a strong argument. Likewise, frequentist methods do not require as much memory to store the results. While these things may seem somewhat trivial, especially with smaller datasets, the fact that Bayesian and Frequentist typically agree in results (especially if you have lots of informative data) means that if you are going to care, you may start caring about the less important things. And of course, if you live in the big data world, these are not trivial at all.

• Non-parametric statistics. I recognize that Bayesian statistics does have non-parametric statistics, but I would argue that the frequentist side of the field has some truly undeniably practical tools, such as the Empirical Distribution Function. No method in the world will ever replace the EDF, nor the Kaplan Meier curves, etc. (although clearly that's not to say those methods are the end of an analysis).

• Less diagnostics. MCMC methods, the most common method for fitting Bayesian models, typically require more work by the user than their frequentist counter part. Usually, the diagnostic for an MLE estimate is so simple that any good algorithm implementation will do it automatically (although that's not to say every available implementation is good...). As such, frequentist algorithmic diagnostics is typically "make sure there's no red text when fitting the model". Given that all statisticians have limited bandwidth, this frees up more time to ask questions like "is my data really approximately normal?" or "are these hazards really proportional?", etc.

• Valid inference under model misspecification. We've all heard that "All models are wrong but some are useful", but different areas of research take this more or less seriously. The Frequentist literature is full of methods for fixing up inference when the model is misspecified: bootstrap estimator, cross-validation, sandwich estimator (link also discusses general MLE inference under model misspecification), generalized estimation equations (GEE's), quasi-likelihood methods, etc. As far as I know, there is very little in the Bayesian literature about inference under model misspecification (although there's a lot of discussion of model checking, i.e., posterior predictive checks). I don't think this just by chance: evaluating how an estimator behaves over repeated trials does not require the estimator to be based on a "true" model, but using Bayes theorem does!

• Freedom from the prior (this is probably the most common reason for why people don't use Bayesian methods for everything). The strength of the Bayesian standpoint is often touted as the use of priors. However, in all of the applied fields I have worked in, the idea of an informative prior in the analysis is not considered. Reading literature on how to elicit priors from non-statistical experts gives good reasoning for this; I've read papers that say things like (cruel straw-man like paraphrasing my own) "Ask the researcher who hired you because they have trouble understanding statistics to give a range that they are 90% certain the effect size they have trouble imagining will be in. This range will typically be too narrow, so arbitrarily try to get them to widen it a little. Ask them if their belief looks like a gamma distribution. You will probably have to draw a gamma distribution for them, and show how it can have heavy tails if the shape parameter is small. This will also involve explaining what a PDF is to them."(note: I don't think even statisticians are really able to accurately say a priori whether they are 90% or 95% certain whether the effect size lies in a range, and this difference can have a substantial effect on the analysis!). Truth be told, I'm being quite unkind and there may be situations where eliciting a prior may be a little more straightforward. But you can see how this is a can of worms. Even if you switch to non-informative priors, it can still be a problem; when transforming parameters, what are easily mistaken for non-informative priors suddenly can be seen as very informative! Another example of this is that I've talked with several researchers who adamantly do not want to hear what another expert's interpretation of the data is because empirically, the other experts tend to be over confident. They'd rather just know what can be inferred from the other expert's data and then come to their own conclusion. I can't recall where I heard it, but somewhere I read the phrase "if you're a Bayesian, you want everyone to be a Frequentist". I interpret that to mean that theoretically, if you're a Bayesian and someone describes their analysis results, you should first try to remove the influence of their prior and then figure out what the impact would be if you had used your own. This little exercise would be simplified if they had given you a confidence interval rather than a credible interval!

Of course, if you abandon informative priors, there is still utility in Bayesian analyses. Personally, this where I believe their highest utility lies; there are some problems that are extremely hard to get any answer from in using MLE methods but can be solved quite easily with MCMC. But my view on this being Bayesian's highest utility is due to strong priors on my part, so take it with a grain of salt.

• (+1) Nice answer, though I'm assuming you meant do not require as much memory to store the results? – jsakaluk Feb 5 '16 at 14:23
• In terms of freedom from priors: are you saying that the less you have to think about and understand your problem, the better? I know several software vendors who would like to talk to you, so you can point-n-click -- or better yet, one-click -- and have an answer to any problem you can imagine! Heck, you don't even need a problem, just feed your data into their website and they'll find all possible problems and solve them, toot sweet ! (Sorry, couldn't resist answering with a cruel straw-man-like comment.) – Wayne Feb 7 '16 at 16:57
• @Wayne: I know you are joking, but that's 100% correct. Statistics is a tool for answering real world problems. I really want to emphasize that it is a tool, not a final product. Regardless of what side of the throughly hashed out "Frequentist vs Bayesian" argument (I sit on "whichever gives me the best answer to my question" side, which means I like both for different problems), there's no arguing that ease of use is a very real utility for any tool. – Cliff AB Feb 7 '16 at 17:13
• Of course, if your tool is frequently producing a terrible product that's a problem. And if I were convinced that a frequentist method was doing this, but a Bayesian method was not, I would quickly endorse the Bayesian method. – Cliff AB Feb 7 '16 at 17:17
• @CliffAB: Ease-of-use is important, and as you say if the results are of equal quality, why choose harder-to-use? At the same time, thinking about, making explicit, and understanding priors (not Bayesian, I mean literally the priors that every scientist, every field, and every study has) is critical to good science. Bayesian statistics is explicit and forces you to think about and understand some of these issues. To the extent that this is not merely pedantic inconvenience, it's arguably good, and so its opposite isn't slam-dunk good either. – Wayne Feb 7 '16 at 17:37

A few concrete advantages of frequentist statistics:

• There are often closed-form solutions to frequentist problems whereas you would need a conjugate prior to have a closed form solution in the Bayesian analogue. This is useful for a number of reasons - one of which is computation time.
• A reason that'll, hopefully, eventually go away: laymen are taught frequentists statistics. If you want to be understood by many, you need to speak frequentist.
• An "Innocent until proven guilty" Null Hypothesis Significance Testing (NHST) approach is useful when the goal is to prove someone wrong (I'm going to assume your right and show the data overwhelming suggests you're wrong). Yes, there are NHST analogues in Bayesian but I find the frequentists versions much more straight-forward and interpretable.
• There is no such thing as a truly uninformative prior which makes some people uncomfortable.
• (+1) Thanks--could you clarify the first point a bit? As someone not well-versed in Bayesian, the point you are making about the need for a "conjugate prior" (?) is lost on me a bit... – jsakaluk Feb 4 '16 at 19:47
• I do not think you are interpreting the frequentist hypothesis test correctly. You just gave $P(H_0\;|\; Data)$, but the p-value is actually $P(Data\;|\;H_0)$. The correct interpretation of the p-value: given the null, there is only an $\alpha$% chance of getting a result as extreme or more extreme then that observed. This misinterpretation is brought up often when arguing for a Bayesian approach. Other than that I like your answer. – Zachary Blumenfeld Feb 4 '16 at 20:05
• @ZacharyBlumenfeld Thanks for pointing that out, I had Bayesian on my mind. Ill fix it now. – TrynnaDoStat Feb 4 '16 at 20:10
• @jsakaluk If the posterior and prior are the same distribution, the prior is said to be conjugate - guaranteeing a closed form posterior. For example, if our data is Bernoulli and we chose a Beta($\alpha$,$\beta$) prior then we know that the posterior is Beta($\alpha + \sum_{i=1}^n x_i$, $\beta + n - \sum_{i=1}^n x_i\!$) without having to do any simulation, sampling, or intense computation. – TrynnaDoStat Feb 4 '16 at 20:24

The most important reason to use Frequentist approaches, which has surprisingly not yet been mentioned, is error control. Very often, research leads to dichotomous interpretations (should I do a study building on this, or not? Should be implement an intervention, or not?). Frequentist approaches allow you to strictly control your Type 1 error rate. Bayesian approaches don't (although some inherit the universal bound from likelihood approaches, but even then, error rates can be quite high in small samples and with relatively low thresholds of evidence (e.g., BF > 3). You can examine Frequentist properties of Bayes factors (see for example http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2604513) but that's still a Frequentist approach. I think very often, researchers care more about error control than about quantifying evidence per se (relative to some specific hypothesis), and I think at the very least, everyone cares about error control to some extent, and thus the two approaches should be used complementarily.

• Good point. I am also thinking of group-sequential methods and other forms of multiple testing, where there seems (from my narrow point of view, which may have overlooked substantial parts of the literature) to have been a lack of interest on the Bayesian side (so far) in terms of getting some kind of error control. Of course in many circumstances Bayesian methods - particularly with somewhat skeptical priors or some kind of shrinkage through a hierarchical model do control errors somewhat to some unquantifiable degree, but a lot more thinking has been done on the frequentist side there. – Björn Feb 6 '16 at 7:30
• (+1) I really like this point...as it is the reason I am philosophically a frequentist....when we do stats to help with inference, then we want our inferences to be more accurate (i.e., less error) than blind guessing. In fact, if I care at all about my inferences being actually true or false (in the sense of being validated by follow-on studies), then error rates are very important. I just cant feel comfortable with Bayesian probability (however, the methods themselves are very useful as sensible "regularlized estimators" for a quantity when sample size is small...think Agresit-Coull) – user75138 Apr 12 '16 at 13:05
• This sounds more like decision theory than bayes/frequentist comparison. Also, with bayesian approach you don't need to worry about stopping rules....I also understand that bayes may achieve a better "balance" between type 1 and type 2 error rates.... – probabilityislogic Dec 9 '17 at 13:50

I think one of the biggest questions, as a statistican, you have to ask yourself is whether or not you believe in, or want to adhere to, the likelihood principle. If you don't believe in the likelihood principle then I think the frequentist paradigm to statistics can be extremely powerful, however, if you do believe in the likelihood principle, then (I believe) you most certainly have to espouse the Bayesian paradigm in or to not violate it.

In case you are unfamiliar with it, what the likelihood principle tells us is the following:

The Likelihood Principle: In making inferences or decisions about $\theta$ after some data $\mathbf{x}$ is observed, all relevant experimental information is contained in the likelihood function: $$\ell(\theta;\mathbf{x})=p(\mathbf{x}|\theta)$$ where $\mathbf{x}$ corresponds to the data observed and is thus fixed.

Furthermore, if $\mathbf{x}$ and $\mathbf{y}$ are two sample points such that $\ell(\theta;\mathbf{x})$ is proportional to $\ell(\theta;\mathbf{y})$, that is, there exists a constant $C(\mathbf{x},\mathbf{y})$ such that

$$\ell(\theta;\mathbf{x})=C(\mathbf{x},\mathbf{y})\ell(\theta;\mathbf{y})\hspace{.1in}\text{for all }\theta,$$

then the conclusions drawn from $\mathbf{x}$ and $\mathbf{y}$ should be identical.\

Note that the constant $C(\mathbf{x},\mathbf{y})$ above may be different for different $(\mathbf{x},\mathbf{y})$ pairs but $C(\mathbf{x},\mathbf{y})$ does not depend on $\theta$.

In the special case of $C(\mathbf{x},\mathbf{y})=1$, the Likelihood Principle states that if two sample points result in the same likelihood function, then they contain the same information about $\theta$. But the Likelihood Principle goes further. It states that even if two sample points have only proportional likelihoods, then they contain equivalent information about $\theta$.

Now, one of the draws of Bayesian statistics is that, under proper priors, the Bayesian paradigm never violates the likelihood principle. However, there are very simple scenarios where the frequentist paradigm will violate the likelihood principle.

Here is a very simple example based on hypothesis testing. Consider the following:

Consider an experiment where 12 Bernoulli trials were run and 3 successes were observed. Depending on the stopping rule we could characterize the data as the following:

• Binomial Distribution: $X|\theta\sim\text{Bin}(n=12,\theta)$ and Data: $x=3$
• Negative Binomial Distribution: $Y|\theta\sim\text{NegBin}(k=3,\theta)$ and Data: $y=12$

And thus we would obtain the following likelihood functions: \begin{align} \ell_1(\theta;x=3)&=\binom{12}{3}\theta^3(1-\theta)^9\\ \ell_2(\theta;y=12)&=\binom{11}{2}\theta^3(1-\theta)^9\\ \end{align} which implies that $$\ell_1(\theta;x)=C(x,y)\ell_2(\theta,y)$$ and thus, by the Likelihood Principle, we should obtain the same inferences about $\theta$ from either likelihood.

Now, imagine testing the following hypotheses from the frequentist paradigm $$H_o:\theta\geq\frac{1}{2}\hspace{.2in}\text{versus}\hspace{.2in}H_a:\theta<\frac{1}{2}$$

For the Binomial model we have the following: \begin{align} \text{p-value}&=P\left(X\leq 3|\theta=\frac{1}{2}\right)\\ &=\binom{12}{0}\left(\frac{1}{2}\right)^{12}+\binom{12}{1} \left(\frac{1}{2}\right)^{12}+ \binom{12}{2}\left(\frac{1}{2}\right)^{12}+\binom{12}{3}\left(\frac{1}{2}\right)^{12}=0.0723 \end{align}

Notice that $\binom{12}{3}\left(\frac{1}{2}\right)^{12}=\ell_1(\frac{1}{2};x=3)$ but the other terms do not satisfy the likelihood principle.

For the Negative Binomial model we have the following: \begin{align} \text{p-value}&=P\left(Y\geq 12|\theta\frac{1}{2}\right)\\ &=\binom{11}{2}\left(\frac{1}{2}\right)^{12}+\binom{12}{2}\left(\frac{1}{2}\right)^{12}+ \binom{13}{2}\left(\frac{1}{2}\right)^{12}+...=0.0375 \end{align}

From the above p-value calculations we see that in the Binomial model we would fail to reject $H_o$ but using the Negative Binomial model we would reject $H_o$. Thus, even though $\ell_1(\theta;x)\propto\ell_2(\theta;y)$ there p-values, and decisions based on these p-values, do not coincide. This p-value argument is one often used by Bayesians against the use of Frequentist p-values.

Now consider again testing the following hypotheses, but from the Bayesian paradigm $$H_o:\theta\geq\frac{1}{2}\hspace{.2in}\text{versus}\hspace{.2in}H_a:\theta<\frac{1}{2}$$

For the Binomial model we have the following: \begin{align} P\left(\theta\geq\frac{1}{2}|x\right)=\int_{1/2}^1\pi(\theta|x)dx=\int_{1/2}^1\theta^3(1-\theta)^9\pi(\theta)d\theta \bigg/\int_{0}^1\theta^3(1-\theta)^9\pi(\theta)d\theta \end{align}

Similarly, for the Negative Binomial model we have the following: \begin{align} P\left(\theta\geq\frac{1}{2}|y\right)=\int_{1/2}^1\pi(\theta|x)dx=\int_{1/2}^1\theta^3(1-\theta)^9\pi(\theta)d\theta \bigg/\int_{0}^1\theta^3(1-\theta)^9\pi(\theta)d\theta \end{align}

Now using Bayesian decision rules, pick $H_o$ if $P(\theta\geq\frac{1}{2}|x)>\frac{1}{2}$ (or some other threshold) and repeat similarly for $y$.

However, $P\left(\theta\geq\frac{1}{2}|x\right)=P\left(\theta\geq\frac{1}{2}|y\right)$ and so we arrive at the same conclusion and thus this approach satisfies the likelihood Principle.

And so to conclude my ramblings, if you don't care about the likelihood principle then being frequentist is great! (If you can't tell, I'm a Bayesian :) )

• I appreciate the clearly thoughtful (and likely time-consuming) response, but I feel like this answer is a bit of a departure from the "answers... conveyed as accessibly as possible..." mandate of the question. – jsakaluk Feb 6 '16 at 3:19
• @jsakaluk I guess what I was aiming for, and wanted to be sure to back up the argument, is that if you are willing to overlook certain things that many applied statisticians take for granted all the time, i.e., the likelihood principle, then using the frequentist paradigm can be a much simpler alternative to the Bayesian paradigm. However, if you cannot then you will most likely have to find alternatives. – RustyStatistician Feb 6 '16 at 8:15
• @RustyStatistician The likelihood principle is a central tenet for likelihoodists. Likelihoodists are not Bayesian at all. I posted links in my answer. Your claim "if you do believe in the likelihood principle, then (I believe) you most certainly have to espouse the Bayesian paradigm" is false. – stan Feb 7 '16 at 13:24
• @Stan I agree with you that yes likelihoodists believe in the likelihood principle sure. But I would find it extremely hard to believe that if you ask any Bayesian if they believe in adhering to the likelihood principle that they would say no they don't (that's just my opinion you don't have to agree). – RustyStatistician Feb 7 '16 at 16:13
• The roles of likelihood principle (LP), conditionality principle (CP) and sufficiency principle (SP) in inference are not simple..this is because these principles relate to evidence (as presented by the data), whereas inference involves going beyond the evidence. This is always risky, but necessary to make progress. See Birnbaums Theorem (discussed here...I don't necessarily agree with the rest of the paper): arxiv.org/abs/1302.5468 – user75138 Apr 12 '16 at 13:29

You and I are both scientists, and as scientists, are chiefly interested in questions of evidence. For that reason, I think Bayesian approaches, when feasible, are preferable.

Bayesian approaches answer our question: What is the strength of evidence for one hypothesis over another? Frequentist approaches, on the other hand, do not: They report only whether the data are weird given one hypothesis.

That said, Andrew Gelman, notable Bayesian, seems to espouse the use of p-values (or p-value-like graphical checks) as a check for errors in model specification. You can see an allusion to this approach in this blog post.

His approach, as I understand it, is something like a two-step process: First, he asks the Bayesian question of what is the evidence for one model over the other. Second, he asks the Frequentist question of whether the preferred model actually looks at all plausible given the data. It seems like a reasonable hybrid approach to me.

• Although the link to the Gelman blog should remain valid, it won't be "today's" after midnight. Edited accordingly. – Nick Cox Feb 4 '16 at 17:51
• I strongly disagree with the notation that frequentist approaches do not measure evidence, and that this is solely in the Bayesian world. You are leaving out the origin of hypothesis testing, such as the LR test, measures the evidence of one hypothesis against the evidence for the other. – Cliff AB Feb 5 '16 at 6:15
• (+1) to @CliffAB - for everyone thinking about "frequentist" statistics, please, please look up "likelihood ratio", "Birnbaum's Theorem", and perhaps read a bit of Royall....don't jump to straw-man arguments involving NHST -- which, by the way, hasn't seemed to dampen scientific progress despite its supposedly catastrophic flaws....that's because statisticians are not carbon-based MINITAB programs...they THINK [yes, doing statistics is actually a profession, just like medicine, or economics, or auto-mechanics,...you can't just read a book, try a formula, and expect truth to land in your lap]. – user75138 Apr 12 '16 at 13:12
• @Bey: Personally, I do believe that p-values have done some dampening to the scientific process (in that biologists are forced to become part time statisticians to publish papers, reducing the time they get to be biologists), but I don't don't think the alternatives to p-values in any way reduce this issue! I feel that the issue of p-values is not their theoretic background, but their ease of use by non-statisticians. Posterior probabilities, (for example) I think make that particular issue worse, rather than better. – Cliff AB Apr 12 '16 at 19:56
• @CliffAB couldn't agree more...didn't think of it from that side..but that's just the nature of publishing I guess...unless research departments can afford to have staff statisticians. Any statistical tool can be misused by one not knowledgeable in its use...pity statistical tools seem so easy to use... – user75138 Apr 12 '16 at 19:58

Personally I'm having difficulty thinking of a situation where the frequentist answer would be preferred over a Bayesian one. My thinking is detailed here and in other blog articles on fharrell.com about problems with p-values and null hypothesis testing. Frequentists tend to ignore a few fundamental problems. Here is just a sample:

• Outside of the Gaussian linear model with constant variance and a few other cases, the p-values that are computed are of unknown accuracy for your dataset and model
• When the experiment is sequential or adaptive, it is often the case that a p-value can't even be computed and one can only set an overall $\alpha$ level to achieve
• Frequentists seem happy to not let the type I error go below, say, 0.05 no matter now the sample size grows
• There is no frequentist prescription for how multiplicity corrections are formed, leading to an ad hoc hodge-podge of methods

Regarding the first point, one commonly used model is the binary logistic model. Its log likelihood is very non-quadratic, and the vast majority of confidence limits and p-values computed for such models are not very accurate. Contrast that with the Bayesian logistic model, which provides exact inference.

Others have mentioned error control as a reason for using frequentist inference. I do not think this is logical, because the error to which they refer is the long-run error, envisioning a process in which thousands of statistical tests are run. A judge who said "the long run false conviction probability in my courtroom is only 0.03" should be disbarred. She is charged with having the highest probability of making the correct decision for the current defendent. On the other hand one minus the posterior probability of an effect is the probabiity of zero or backwards effect and is the error probability we actually need.

• "There is no frequentist prescription for how multiplicity corrections are formed, leading to an ad hoc hodge-podge of methods." On the other hand, I've never seen a Bayesian do multiplicity corrections at all. Andrew Gelman even proudly declares that he never uses them. For instance, I've seen people report marginal 95% credible intervals for $\theta_1, \ldots, \theta_k$, but the joint credibility of those $k$ intervals is not 95%. Nor is it obvious how best to address this. Do you have any advice or examples? – civilstat Feb 20 '19 at 20:13

Many people do not seem aware of a third philosophical school: likelihoodism. AWF Edwards's book, Likelihood, is probably the best place to read up on it. Here is a short article he wrote.
Likelihoodism eschews p-values, like Bayesianism, but also eschews the Bayesian's often dubious prior. There is an intro treatment here as well.

• There's algorithmic probability approach by Vovk, developed from Kolmogorov's ideas. – Aksakal Feb 4 '16 at 19:18
• "Many people do not seem aware of a third philosophical school: likelihoodism" I do not think that this sentence is true in 2016... – Tim Feb 4 '16 at 21:28
• @Tim, although everybody I know is familiar with frequentism and Bayesianism, I have never met anyone who had heard of likelihoodism. The original questioner seems to be like my colleagues who were trained in frequentism and are becoming increasingly interested in Bayesianism. Perhaps most people who read my answer above think I am referring to maximum likelihood estimation or testing hypotheses using likelihood ratios. Nope! I suggest Yudi Pawitan and this lecture – stan Feb 5 '16 at 8:06
• None of those approaches is religion, so there is not much to believe, they are just helpful for certain kind of problems, and some of the approaches are better suited for some problems and other for others :) – Tim Feb 5 '16 at 8:50
• (+1) for mentioning likelihood school and for the comment regarding Pawitan. Pawitan's book "In All Likelihood" dramatically widened and enhanced by statistical practice...I was also only aware of Bayes vs Frequentism. He tackles a lot of philosophical and methodological aspects of Bayes, "classical" frequentism, and, of course, covers the pure likelihood school. Just a great book for becoming a more sophisticated user of stats...regardless of your philosophical inclinations. – user75138 Apr 12 '16 at 13:19

One of the biggest disadvantages of frequentist approaches to model building has always been, as TrynnaDoStats notes in his first point, the challenges involved with inverting big closed-form solutions. Closed-form matrix inversion requires that the entire matrix be resident in RAM, a significant limitation on single CPU platforms with either large amounts of data or massively categorical features. Bayesian methods have been able to work around this challenge by simulating random draws from a specified prior. This has always been one of the biggest selling points of Bayesian solutions, although answers are obtained only at a significant cost in CPU.

Andrew Ainslie and Ken Train, in a paper from about 10 years ago that I have lost the reference to, compared finite mixture (which are frequentist or closed form) with Bayesian approaches to model-building and found that across a wide range of functional forms and performance metrics, the two methods delivered essentially equivalent results. Where Bayesian solutions had an edge or possessed greater flexibility were in those instances where the information was both sparse and very high-dimensional.

However, that paper was written before "divide and conquer" algorithms were developed that leverage massively parallel platforms, e.g., see Chen and Minge's paper for more about this http://dimacs.rutgers.edu/TechnicalReports/TechReports/2012/2012-01.pdf

The advent of D&C approaches has meant that, even for the hairiest, sparsest, most high dimensional problems, Bayesian approaches no longer have an advantage over frequentist methods. The two methods are at parity.

This relatively recent development is worth noting in any debate about the practical advantages or limitations of either method.

• I think this is a nice addition to the discussion (+1) but I find it hard to follow. It really, really, really postpones its punch-line... Maybe you could reorganise it a bit? :) – usεr11852 Feb 5 '16 at 18:35
• @user11852 You don't say that the post fails to communicate something useful whereas you do find the development of the logic not up to journalistic standards. Since this thread has gone "community," I'm not too inclined (motivated?) to work on reorganizing it around your suggestion. It can stand as is. But thank you anyway for the upvote and comment. – Mike Hunter Feb 5 '16 at 19:17
• 1.) Matrix inversion is often used for MLE estimation (which is only one of many frequentist methods), but not always. My work in MLE estimation involves optimization over often up to $n$ parameters (i.e. parameter space can grow linearly with sample size) and matrix inversion is absolutely not an option...but I still optimize the likelihood! 2.) Matrix inversion still happens all the time in Bayesian statistics, such as a block updater sampler. – Cliff AB Feb 7 '16 at 17:24
• @CliffAB I was thinking of ANOVA-type inversion of the matrix of cross-products. – Mike Hunter Feb 16 '16 at 15:13
• @DJohnson: I see. But my point was that matrix inversion is orthogonal to frequentist vs bayesian methods; both camps use tools that do something very similar (at least in terms of computational costs) in many of their methods. – Cliff AB Feb 16 '16 at 17:37

Frequentist tests focus on falsifying the null hypothesis. However, Null Hypothesis Significance Testing (NHST) can also be done from a Bayesian perspective, because in all cases NHST is simply a calculation of P( Observed Effect | Effect = 0 ). So, it's hard to identify a time when it would be necessary to conduct NHST from a frequentist perspective.

That being said, the best argument for conducting NHST using a frequentist approach is ease and accessibility. People are taught frequentist statistics. So, it's easier to run a frequentist NHST, because there are many more statistical packages that make it simple to do this. Similarly, it is easier to communicate the results of a frequentist NHST, because people are familiar with this form of NHST. So, I see that as the best argument for frequentist approaches: accessibility to stats programs that will run them and ease of communication of results to colleagues. This is just cultural, though, so this argument could change if frequentist approaches lose their hegemony.

• The comments about what Fisher thought seem overwrought here unless you can provide exact quotations. The null hypothesis is a device as a part of a significance test to try to discourage scientists from over-interpreting results from small samples. Fisher was as keen as anybody else that scientists should use statistics to do good science; he was himself a very serious contributor to genetics. – Nick Cox Feb 4 '16 at 17:56
• I agree completely, and so I edited the answer to remove the speculation about Fisher's mental state. – Liz Page-Gould Feb 4 '16 at 18:38

• The fundamental difference between the bayesian and frequentist statistician is that the bayesian is willing to extend the tools of probability to situations where the frequentist wouldn't.

• More specifically, the bayesian is willing to use probability to model the uncertainty in her own mind over various parameters. To the frequentist, these parameters are scalars (albeit scalars where the statistician does not know the true value). To the Bayesian, various parameters are represented as random variables! This is extremely different. The Bayesian's uncertainty over parameters valeus is represented by a prior.
• In Bayesian statistics, the hope is that after observing data, the posterior overwhelms the prior, that the prior doesn't matter. But this often isn't the case: results may be sensitive to the choice of prior! Different Bayesians with different priors need not agree on the posterior.

A key point to keep in mind is that statements of the frequentist statistician are statements that any two Bayesians can agree on, regardless of their prior beliefs!

The frequentist does not comment on priors or posteriors, merely the likelihood.

The statements of the frequentist statistician in some sense are less ambitious, but the bolder statements of the Bayesian can significantly rely on the assignment of a prior. In situations where priors matter and where there is disagreement on priors, the more limited, conditional statements of frequentist statistics may stand on firmer ground.

The goal of much research is not to reach a final conclusion, but just to obtain a little more evidence to incrementally push the community's sense of a question in one direction.

Bayesian statistics are indispensable when what you need is to evaluate a decision or conclusion in light of the available evidence. Quality control would be impossible without Bayesian statistics. Any procedure where you need to take some data and then act on it (robotics, machine learning, business decision making) benefits from Bayesian statistics.

But a lot of researchers are not doing that. They are running some experiments, collecting some data, and then saying "The data points this way", without really worrying too much about whether that's the best conclusion given all the evidence others have gathered so far. Science can be a slow process, and a statement like "The probability that this model is correct is 72%!" is often premature or unnecessary.

This is appropriate in a simple mathematical way, too, because frequentist statistics often turn out to be mathematically the same as the update-step of a Bayesian statistic. In other words, while Bayesian statistics is (Prior Model, Evidence) → New Model, frequentist statistics is just Evidence, and leaves it to others to fill in the other two parts.

• Although much of this post is interesting, it consists of many unsupported opinions. Please consult our help center concerning what kinds of answers are expected on this site. – whuber Feb 5 '16 at 15:03
• @whuber I see. I have added one citation I can remember off the top of my head, but the rest I don't have citations for, so if it seems too unsupported I can delete it. – Owen Feb 5 '16 at 15:08
• I'm surprised that you mentioned quality control, since it seems like an area where the frequentist interpretation of probability (relative frequency over many trials) would be very natural: given that the factory is working correctly, how likely are we to see this many (or more) broken widgets? Could I push you to elaborate on what makes Bayesian statistics particularly useful for QC? – Matt Krause Feb 5 '16 at 20:35
• @MattKrause Suppose our goal is to ship defective widgets at a rate <1%. We know the factory produces defective widgets at a rate of 10%, and we have a test whose Type-I and Type-II error rates are s and 1/(sqrt(4 - 1/s^2)) where s is a strictness parameter. What should we use for the strictness? – Owen Feb 5 '16 at 22:20
• The idea that frequentist statistics cannot not combine information from successive studies seems to ignore the field of meta-analyses. – Cliff AB Jun 8 '16 at 23:03

The actual execution of a Bayesian method is more technical than that of a Frequentist. By "more technical" I mean things like: 1) choosing priors, 2) programming your model in a BUGS/JAGS/STAN, and 3) thinking about sampling and convergence.

Obviously, #1 is pretty much not optional, by definition of Bayesian. Though with some problems and procedures, there can be reasonable defaults, somewhat hiding the issue from the user. (Though this can also cause problems!)

Whether #2 is an issue depends on the software you use. Bayesian statistics has a bent towards more general solutions than frequentist statistical methods, and tools like BUGS, JAGS, and STAN are a natural expression of this. However, there are Bayesian functions in various software packages that appear to work like the typical frequentist procedure, so this is not always an issue. (And recent solutions like the R packages rstanarm and brms are bridging this gap.) Still, using these tools is very similar to programming in a new language.

Item #3 is usually applicable, since the majority of real-world Bayesian applications are going to use MCMC sampling. (On the other hand, frequentist MLE-based procedures use optimization which may converge to a local minima or not converge at all, and I wonder how many users should be checking this and don't?)

As I said in a comment, I'm not sure that freedom from priors is actually a scientific benefit. It's certainly convenient in several ways and at several points in the publication process, but I'm not sure it actually makes for better science. (And in the big picture, we all have to be aware of our priors as scientists, or we'll suffer from all kinds of biases in our investigations, regardless of what statistical methods we use.)

• In regards to (3), many classical stats models (i.e. glm's) have concave log-likelihoods, so it's very rare that standard algorithms should fail, outside of extreme corner cases. In regards to non-concave problems (i.e. NNs), while these do require heavy concern about improper convergence (which is typically understood by users), these are (not coincidentially) also problems in which classic MCMC algorithms will horribly fail if only run for, say, one human's lifetime. However, it's generally less of a stretch to fix up the MCMC than the optimization algorithm! – Cliff AB Oct 5 '17 at 21:27

Conceptually: I don't know. I believe Bayesian statistics is the most logical way to think but I coudn't justify why.

The advantage of frequentist is that it is easier for most people at elementary level. But for me it was strange. It took years until I could really clarify intellectually what a confidence interval is. But when I started facing practical situations, frequentist ideas appeared to be simple and highly relevant.

Empirically

The most important question I try to focus on nowadays is more about practical efficiency: personal work time, precision, and computation speed.

Personal work time: For basic questions, I actually almost never use Bayesian methods: I use basic frequentist tools and will always prefer a t-test over a Bayesian equivalent that would just give me a headache. When I want to know if I'm significantly better at tictactoe than my girlfriend, I do a chi-squared :-). Actually, even in serious work as a computer scientist, frequentist basic tools are just invaluable to investigate problems and avoid false conclusions due to random.

Precision: In machine learning where prediction matters more than analysis, there is not an absolute boundary between Bayesian and frequentist. MLE is a frequentist approcah: just an estimator. But regularized MLE (MAP) is a partially Bayesian approach: you find the mode of the posterior and you don't care for the rest of the posterior. I don't know of a frequentist justification of why use regularization. Practically, regularization is sometimes just inevitable because the raw MLE estimate is so overfitted that 0 would be a better predictor. If regularization is agreed to be a truly Bayesian method, then this alone justifies that Bayes can learn with less data.

Computation speed: frequentist methods are most often computationally faster and simpler to implement. And somehow regularization provides a cheap way to introduce a bit of Bayes in them. It might be because Bayesian methods are still not as optimized as they could. For example, some LDA implementations are fast nowadays. But they required very hard work. For entropy estimations, the first advanced methods were Bayesian. They worked great but soon frequentist methods were discovered and take much less computation time... For computation time frequentist methods are generally clearly superior. It is not absurd, if your are a Bayesian, to think of frequentist methods as approximations of Bayesian methods.

• "I don't know of a frequentist justification of why [to] use regularization". That's easy; under repeated trials, it has shown to lower out-of-sample error. – Cliff AB Feb 3 '19 at 4:20

One type of problem in which a particular Frequentist based approach has essentially dominated any Bayesian is that of prediction in the M-open case.

What does M-open mean?

M-open implies that the true model that generates the data does not appear in the set of models we are considering. For example, if the true mean of $$y$$ is quadratic as a function of $$x$$, yet we only consider models with the mean a linear function of $$x$$, we are in the M-open case. In other words, model miss-specification results in an M-open case.

In most cases, this is a huge problem for Bayesian analyses; pretty much all theory that I know about relies on the model being correctly specified. Of course, as critical statisticians, we should think that our model is always misspecified. This is quite an issue; most of our theory is based on the model being correct, yet we know it never is. Basically, we're just crossing our fingers hoping that our model is not too incorrect.

Why do Frequentist methods handle this better?

Not all do. For example, if we use standard MLE tools for creating the standard errors or building prediction intervals, we're not better off than using Bayesian methods.

However, there is one particular Frequentist tool that is very specifically intended for exactly this purpose: cross validation. Here, in order to estimate how well our model will predict on new data, we simply leave of some of the data when fitting the model and measure how well our model predicts the unseen data.

Note that this method is completely ambivalent to model miss-specification, it merely provides a method for us to estimate for how well a model will predict on new data, regardless of whether the model is "correct" or not.

I don't think it's too hard to argue that this really changes the approach to predictive modeling that's hard to justify from a Bayesian perspective (prior is supposed to represent prior knowledge before seeing data, likelihood function is the model, etc.) to one that's very easy to justify from a Frequentist perspective (we chose the model + regularization parameters that, over repeated sampling, leads to the best out of sample errors).

This has completely revolutionized how predictive inference is done. I don't think any statistician would (or at least, should) seriously consider a predictive model that wasn't built or checked with cross-validation, when it's available (i.e., we can reasonable assume observations are independent, not trying to account for sampling bias, etc.).