When are Bayesian methods preferable to Frequentist?

Question

I really want to learn about Bayesian techniques, so I have been trying to teach myself a bit. However, I am having a hard time seeing when using Bayesian techniques ever confer an advantage over Frequentist methods. For example: I've seen in the literature a bit about how some use informative priors whereas others use non-informative prior. But if you're using a non-informative prior (which seems really common?) and you find that the posterior distribution is, say, a beta distribution...couldn't you have just fit a beta distribution in the beginning and called it good? I don't see how constructing a prior distribution that tells you nothing...can, well, really tell you anything?

It does turn out that some methods I have been using in R use a mixture of Bayesian and Frequentist methods (the authors acknowledge this is somewhat inconsistent) and I cannot even discern what parts are Bayesian. Aside from distribution fitting, I can't even figure out HOW you would use Bayesian methods. Is there "Bayesian regression"? What would that look like? All I can imagine is guessing at the underlying distribution over and over again while the Frequentist thinks about the data some, eyeballs it, sees a Poisson distribution and runs a GLM. (This isn't a criticism...I really just don't understand!)

So..maybe some elementary examples would help? And if you know of some practical references for real beginners like myself, that would be really helpful too!

Answer 1

Here are some links which may interest you comparing frequentist and Bayesian methods:

• http://www.stat.ufl.edu/archived/casella/Talks/BayesRefresher.pdf
  • Archived here: https://web.archive.org/web/20140308021414/https://stat.ufl.edu/archived/casella/Talks/BayesRefresher.pdf
• http://www.bayesian-inference.com/advantagesbayesian
• http://www.researchgate.net/post/Bayesian_vs_frequentist_statistics2

In a nutshell, the way I have understood it, given a specific set of data, the frequentist believes that there is a true, underlying distribution from which said data was generated. The inability to get the exact parameters is a function of finite sample size. The Bayesian, on the other hand, think that we start with some assumption about the parameters (even if unknowingly) and use the data to refine our opinion about those parameters. Both are trying to develop a model which can explain the observations and make predictions; the difference is in the assumptions (both actual and philosophical). As a pithy, non-rigorous, statement, one can say the frequentist believes that the parameters are fixed and the data is random; the Bayesian believes the data is fixed and the parameters are random. Which is better or preferable? To answer that you have to dig in and realize just what assumptions each entails (e.g. are parameters asymptotically normal?).

Answer 2

One of many interesting aspects of the contrasts between the two approaches is that it is very difficult to have formal interpretation for many quantities we obtain in the frequentist domain. One example is the ever-increasing importance of penalization methods (shrinkage). When one obtains penalized maximum likelihood estimates, the biased point estimates and "confidence intervals" are very difficult to interpret. On the other hand, the Bayesian posterior distribution for parameters that are penalized towards zero using a prior distribution concentrated around zero have completely standard interpretations.

Answer 3

I'm stealing this wholesale from the Stan users group. Michael Betancourt provided this really good discussion of identifiability in Bayesian inference, which I believe bears on your request for a contrast of the two statistical schools.

The first difference with a Bayesian analysis will be the presence of priors which, even when weak, will constrain the posterior mass for those 4 parameters into a finite neighborhood (otherwise you wouldn't have had a valid prior in the first place). Despite this, you can still have non-identifiability in the sense that the posterior will not converge to a point mass in the limit of infinite data. In a very real sense, however, that doesn't matter because (a) the infinite data limit isn't real anyways and (b) Bayesian inference doesn't report point estimates but rather distributions. In practice such non-identifiability will result in large correlations between the parameters (perhaps even non-convexity) but a proper Bayesian analysis will identify those correlations. Even if you report single parameter marginals you'll get distributions that span the marginal variance rather than the conditional variance at any point (which is what a standard frequentist result would quote, and why identifiability is really important there), and it's really the marginal variance that best encodes the uncertainty regarding a parameter.

Simple example: consider a model with parameters $\mu_1$ and $\mu_2$ with likelihood $\mathcal{N}(x | \mu_1 + \mu_2, \sigma)$. No matter how much data you collect, the likelihood will not converge to a point but rather a line $\mu_1 + \mu_2 = 0$. The conditional variance of $\mu_1$ and $\mu_2$ at any point on that line will be really small, despite the fact that the parameters can't really be identified.

Bayesian priors constrain the posterior distribution from that line to a long, cigar shaped distribution. Not easily to sample from but at least compact. A good Bayesian analysis will explore the entirety of that cigar, either identifying the correlation between $\mu_1$ and $\mu_2$ or returning the marginal variances that correspond to the projection of the long cigar onto the $\mu_1$ or $\mu_2$ axes, which give a much more faithful summary of the uncertainty in the parameters than the conditional variances.

Answer 4

The key difference between Bayesian and frequentist approaches lies in the definition of a probability, so if it is necessary to treat probabilties strictly as a long run frequency then frequentist approaches are reasonable, if it isn't then you should use a Bayesian approach. If either interpretation is acceptable, then Bayesian and frequentist approaches are likely to be reasonable.

Another way of putting it, is if you want to know what inferences you can draw from a particular experiment, you probably want to be Bayesian; if you want to draw conclusions about some population of experiments (e.g. quality control) then frequentist methods are well suited.

Essentially, the important thing is to know what question you want answered, and choose the form of analysis that answers the question most directly.

Answer 5

The Bayesian approach to hypothesis testing is a lot more intuitive. Suppose you wish to calculate a two-sample T-test. You can use the T-statistic to do this and then find the p-value. One can argue this is not a very intuitive method, it is very ad hoc, and very specific to the problem. The Bayesian approach is to simply treat the two parameters as distributions. Once you feed the data into your problem you will get your posterior distributions. Now you can run some simulations to see how frequently your hypothesis is true in the space-of-all-possibilities. Interesting, the probability you will get from this method will be consistent with the non-Bayesian t-test method. However, this Bayesian approach is more intuitive to understand and can easily generalize to hypothesis tests which violate the assumptions of the t-test.

Answer 6

Imagine you estimate a probability that AC Milan beats Real Madrid and they have played 3 games in their current line-up. All 3 games were won by Real Madrid. Then a frequentist says that Milan can never beat Real Madrid, which makes no sense. A Bayesian might take a prior from the previous seasons, which would result in some positive posterior probability for Milan.