# When are Bayesian methods preferable to Frequentist?

I really want to learn about Bayesian techniques, so I have been trying to teach myself a bit. However, I am having a hard time seeing when using Bayesian techniques ever confer an advantage over Frequentist methods. For example: I've seen in the literature a bit about how some use informative priors whereas others use non-informative prior. But if you're using a non-informative prior (which seems really common?) and you find that the posterior distribution is, say, a beta distribution...couldn't you have just fit a beta distribution in the beginning and called it good? I don't see how constructing a prior distribution that tells you nothing...can, well, really tell you anything?

It does turn out that some methods I have been using in R use a mixture of Bayesian and Frequentist methods (the authors acknowledge this is somewhat inconsistent) and I cannot even discern what parts are Bayesian. Aside from distribution fitting, I can't even figure out HOW you would use Bayesian methods. Is there "Bayesian regression"? What would that look like? All I can imagine is guessing at the underlying distribution over and over again while the Frequentist thinks about the data some, eyeballs it, sees a Poisson distribution and runs a GLM. (This isn't a criticism...I really just don't understand!)

So..maybe some elementary examples would help? And if you know of some practical references for real beginners like myself, that would be really helpful too!

• Possible duplicate of this? Jun 17, 2014 at 3:21
• Er, looks like? Welcome to close since that comes close to answering my question. I still wonder about the simpler situations I described (since I've never heard of the techniques listed on that thread) but I suppose my answer is that people DON'T use bayesian techniques for regression/etc because established and easy frequentist techniques exist? Jun 17, 2014 at 4:10
• People do use Bayesian techniques for regression. But because the frequentist methods are very convenient and many people are pragmatic about which approach they use, so often people who are happy to use either will use ordinary regression if there's no need for something more complicated. But as soon as you need to deal with a bit more complexity, or to formally incorporate prior information, or any number of other reasons then the modest additional work in Bayesian approaches start to look good. Jun 17, 2014 at 4:18
• Something else that's relevant.. for regression in a Bayesian setting, the most used priors for the coefficients are the multivariate Normal and the multivariate Laplace. Using these priors works out to putting shrinkage penalties on the coefficients, making them equivalent to using ridge regression or the LASSO, respectively, if one were to take the MAP estimate of the coefficients after a Bayesian algorithm. It's much more economical to calculate these results in a way that isn't fully Bayesian, and if they're basically equivalent.. why bother?
– user44764
Jun 17, 2014 at 12:37
• (2) that is why bother unless you specifically want to do Bayesian inference on the coefficients?
– user44764
Jun 17, 2014 at 12:46

## 6 Answers

Here are some links which may interest you comparing frequentist and Bayesian methods:

In a nutshell, the way I have understood it, given a specific set of data, the frequentist believes that there is a true, underlying distribution from which said data was generated. The inability to get the exact parameters is a function of finite sample size. The Bayesian, on the other hand, think that we start with some assumption about the parameters (even if unknowingly) and use the data to refine our opinion about those parameters. Both are trying to develop a model which can explain the observations and make predictions; the difference is in the assumptions (both actual and philosophical). As a pithy, non-rigorous, statement, one can say the frequentist believes that the parameters are fixed and the data is random; the Bayesian believes the data is fixed and the parameters are random. Which is better or preferable? To answer that you have to dig in and realize just what assumptions each entails (e.g. are parameters asymptotically normal?).

• Lots of great and interesting answers, but this answered my questions the most directly. Thanks Jun 19, 2014 at 16:40
• @Avraham the first link is broken Dec 23, 2019 at 10:28
• @ErikHambardzumyan Thanks. Found a version saved on the wayback machine. Dec 25, 2019 at 16:10

One of many interesting aspects of the contrasts between the two approaches is that it is very difficult to have formal interpretation for many quantities we obtain in the frequentist domain. One example is the ever-increasing importance of penalization methods (shrinkage). When one obtains penalized maximum likelihood estimates, the biased point estimates and "confidence intervals" are very difficult to interpret. On the other hand, the Bayesian posterior distribution for parameters that are penalized towards zero using a prior distribution concentrated around zero have completely standard interpretations.

• This is a good point. I wonder if it is primarily true when lambda is selected a-priori, though. Often, one might select lambda using cross-validation to optimize the out of sample prediction error. In which case, it strikes me as weird to say that the lambda is equivalent to the 'prior information' that you brought to the analysis. Jun 17, 2014 at 16:12
• If the penalty is quadratic this is equivalent to a Gaussian prior with mean zero and I believe $\lambda = \sigma^{-2}$. [Don't use prediction error to optimize; use penalized log-likelihood or effective AIC.] Frequentists don't typically know how to account for uncertainty in $\lambda$. Jun 17, 2014 at 16:36
• I'd say Lambda was a hyper-parameter of the prior (for which being more Bayesian you could have a hyper-prior and marginalize over that as well jmlr.org/papers/volume8/cawley07a/cawley07a.pdf) Jun 17, 2014 at 17:13
• My understanding is penalized regressions are best used for prediction or machine learning models, where we're less interested in confidence intervals anyways. For models where we seek unbiased estimates of parameters, we won't be using penalized regression methods and conventional 95% confidence intervals are more appropriate. Dec 15, 2023 at 20:00
• Bias is with respect to what estimates you get upon repeated sampling (as if you could actually re-run an experiment in an identical fashion) is particular to the frequentist school of thought. Bayes has a wholly different consideration: recovery of the true parameter that generated these data. And a beauty of Bayes is that penalization methods using shrinkage priors are used not only for prediction with high-dimensional data but also in providing skeptical priors for individual effects. Dec 16, 2023 at 14:09

I'm stealing this wholesale from the Stan users group. Michael Betancourt provided this really good discussion of identifiability in Bayesian inference, which I believe bears on your request for a contrast of the two statistical schools.

The first difference with a Bayesian analysis will be the presence of priors which, even when weak, will constrain the posterior mass for those 4 parameters into a finite neighborhood (otherwise you wouldn't have had a valid prior in the first place). Despite this, you can still have non-identifiability in the sense that the posterior will not converge to a point mass in the limit of infinite data. In a very real sense, however, that doesn't matter because (a) the infinite data limit isn't real anyways and (b) Bayesian inference doesn't report point estimates but rather distributions. In practice such non-identifiability will result in large correlations between the parameters (perhaps even non-convexity) but a proper Bayesian analysis will identify those correlations. Even if you report single parameter marginals you'll get distributions that span the marginal variance rather than the conditional variance at any point (which is what a standard frequentist result would quote, and why identifiability is really important there), and it's really the marginal variance that best encodes the uncertainty regarding a parameter.

Simple example: consider a model with parameters $\mu_1$ and $\mu_2$ with likelihood $\mathcal{N}(x | \mu_1 + \mu_2, \sigma)$. No matter how much data you collect, the likelihood will not converge to a point but rather a line $\mu_1 + \mu_2 = 0$. The conditional variance of $\mu_1$ and $\mu_2$ at any point on that line will be really small, despite the fact that the parameters can't really be identified.

Bayesian priors constrain the posterior distribution from that line to a long, cigar shaped distribution. Not easily to sample from but at least compact. A good Bayesian analysis will explore the entirety of that cigar, either identifying the correlation between $\mu_1$ and $\mu_2$ or returning the marginal variances that correspond to the projection of the long cigar onto the $\mu_1$ or $\mu_2$ axes, which give a much more faithful summary of the uncertainty in the parameters than the conditional variances.

The key difference between Bayesian and frequentist approaches lies in the definition of a probability, so if it is necessary to treat probabilties strictly as a long run frequency then frequentist approaches are reasonable, if it isn't then you should use a Bayesian approach. If either interpretation is acceptable, then Bayesian and frequentist approaches are likely to be reasonable.

Another way of putting it, is if you want to know what inferences you can draw from a particular experiment, you probably want to be Bayesian; if you want to draw conclusions about some population of experiments (e.g. quality control) then frequentist methods are well suited.

Essentially, the important thing is to know what question you want answered, and choose the form of analysis that answers the question most directly.

The Bayesian approach to hypothesis testing is a lot more intuitive. Suppose you wish to calculate a two-sample T-test. You can use the T-statistic to do this and then find the p-value. One can argue this is not a very intuitive method, it is very ad hoc, and very specific to the problem. The Bayesian approach is to simply treat the two parameters as distributions. Once you feed the data into your problem you will get your posterior distributions. Now you can run some simulations to see how frequently your hypothesis is true in the space-of-all-possibilities. Interesting, the probability you will get from this method will be consistent with the non-Bayesian t-test method. However, this Bayesian approach is more intuitive to understand and can easily generalize to hypothesis tests which violate the assumptions of the t-test.

Imagine you estimate a probability that AC Milan beats Real Madrid and they have played 3 games in their current line-up. All 3 games were won by Real Madrid. Then a frequentist says that Milan can never beat Real Madrid, which makes no sense. A Bayesian might take a prior from the previous seasons, which would result in some positive posterior probability for Milan.

• This is a straw man argument. I am unaware of any classical ("frequentist") procedure that would conclude it is impossible for M to beat RM.
– whuber
Jan 26, 2021 at 14:25
• @whuber Well, the frequentist Maximum Likelihood method would estimate the probability of Real Madrid winning as 3 / 3 = 1, right? It would say that the sample proportion is the best estimate of the true proportion. Jan 26, 2021 at 14:28
• Classical procedures include far more techniques than ML. Nobody would use ML on a dataset like this (except possibly as a textbook illustration of the calculations). Moreover, a correct application of ML would include an analysis of the uncertainty in the estimate, which would preclude making the obviously bad conclusion you attribute to this method.
– whuber
Jan 26, 2021 at 15:16
• This is exactly what an uneducated researcher might do ;-). In your example there isn't enough data to justify using ML. In either example, the ML estimate is on the boundary of the parameter space, which automatically urges caution and further examination on the analyst. None of this inexorably "leads to Bayesian analysis." In fact, such bad arguments just create division among statisticians where none is needed.
– whuber
Jan 26, 2021 at 15:26
• Because you continue to conflate "frequentist" with mindless application of ML, we certainly disagree. I find this a fruitless conversation and need to move on to more productive work.
– whuber
Jan 26, 2021 at 15:33