16
$\begingroup$

I really want to learn about Bayesian techniques, so I have been trying to teach myself a bit. However, I am having a hard time seeing when using Bayesian techniques ever confer an advantage over Frequentist methods. For example: I've seen in the literature a bit about how some use informative priors whereas others use non-informative prior. But if you're using a non-informative prior (which seems really common?) and you find that the posterior distribution is, say, a beta distribution...couldn't you have just fit a beta distribution in the beginning and called it good? I don't see how constructing a prior distribution that tells you nothing...can, well, really tell you anything?

It does turn out that some methods I have been using in R use a mixture of Bayesian and Frequentist methods (the authors acknowledge this is somewhat inconsistent) and I cannot even discern what parts are Bayesian. Aside from distribution fitting, I can't even figure out HOW you would use Bayesian methods. Is there "Bayesian regression"? What would that look like? All I can imagine is guessing at the underlying distribution over and over again while the Frequentist thinks about the data some, eyeballs it, sees a Poisson distribution and runs a GLM. (This isn't a criticism...I really just don't understand!)

So..maybe some elementary examples would help? And if you know of some practical references for real beginners like myself, that would be really helpful too!

$\endgroup$
  • $\begingroup$ Possible duplicate of this? $\endgroup$ – Glen_b Jun 17 '14 at 3:21
  • $\begingroup$ Er, looks like? Welcome to close since that comes close to answering my question. I still wonder about the simpler situations I described (since I've never heard of the techniques listed on that thread) but I suppose my answer is that people DON'T use bayesian techniques for regression/etc because established and easy frequentist techniques exist? $\endgroup$ – HFBrowning Jun 17 '14 at 4:10
  • 2
    $\begingroup$ People do use Bayesian techniques for regression. But because the frequentist methods are very convenient and many people are pragmatic about which approach they use, so often people who are happy to use either will use ordinary regression if there's no need for something more complicated. But as soon as you need to deal with a bit more complexity, or to formally incorporate prior information, or any number of other reasons then the modest additional work in Bayesian approaches start to look good. $\endgroup$ – Glen_b Jun 17 '14 at 4:18
  • $\begingroup$ That makes sense, thank you. Reading around on some of the other threads has clarified the uses for me as well. $\endgroup$ – HFBrowning Jun 17 '14 at 4:39
  • $\begingroup$ Something else that's relevant.. for regression in a Bayesian setting, the most used priors for the coefficients are the multivariate Normal and the multivariate Laplace. Using these priors works out to putting shrinkage penalties on the coefficients, making them equivalent to using ridge regression or the LASSO, respectively, if one were to take the MAP estimate of the coefficients after a Bayesian algorithm. It's much more economical to calculate these results in a way that isn't fully Bayesian, and if they're basically equivalent.. why bother? $\endgroup$ – user44764 Jun 17 '14 at 12:37
9
$\begingroup$

Here are some links which may interest you comparing frequentist and Bayesian methods:

In a nutshell, the way I have understood it, given a specific set of data, the frequentist believes that there is a true, underlying distribution from which said data was generated. The inability to get the exact parameters is a function of finite sample size. The Bayesian, on the other hand, think that we start with some assumption about the parameters (even if unknowingly) and use the data to refine our opinion about those parameters. Both are trying to develop a model which can explain the observations and make predictions; the difference is in the assumptions (both actual and philosophical). As a pithy, non-rigorous, statement, one can say the frequentist believes that the parameters are fixed and the data is random; the Bayesian believes the data is fixed and the parameters are random. Which is better or preferable? To answer that you have to dig in and realize just what assumptions each entails (e.g. are parameters asymptotically normal?).

$\endgroup$
  • 2
    $\begingroup$ Lots of great and interesting answers, but this answered my questions the most directly. Thanks $\endgroup$ – HFBrowning Jun 19 '14 at 16:40
8
$\begingroup$

One of many interesting aspects of the contrasts between the two approaches is that it is very difficult to have formal interpretation for many quantities we obtain in the frequentist domain. One example is the ever-increasing importance of penalization methods (shrinkage). When one obtains penalized maximum likelihood estimates, the biased point estimates and "confidence intervals" are very difficult to interpret. On the other hand, the Bayesian posterior distribution for parameters that are penalized towards zero using a prior distribution concentrated around zero have completely standard interpretations.

$\endgroup$
  • 1
    $\begingroup$ This is a good point. I wonder if it is primarily true when lambda is selected a-priori, though. Often, one might select lambda using cross-validation to optimize the out of sample prediction error. In which case, it strikes me as weird to say that the lambda is equivalent to the 'prior information' that you brought to the analysis. $\endgroup$ – gung Jun 17 '14 at 16:12
  • 1
    $\begingroup$ If the penalty is quadratic this is equivalent to a Gaussian prior with mean zero and I believe $\lambda = \sigma^{-2}$. [Don't use prediction error to optimize; use penalized log-likelihood or effective AIC.] Frequentists don't typically know how to account for uncertainty in $\lambda$. $\endgroup$ – Frank Harrell Jun 17 '14 at 16:36
  • 3
    $\begingroup$ I'd say Lambda was a hyper-parameter of the prior (for which being more Bayesian you could have a hyper-prior and marginalize over that as well jmlr.org/papers/volume8/cawley07a/cawley07a.pdf) $\endgroup$ – Dikran Marsupial Jun 17 '14 at 17:13
5
$\begingroup$

I'm stealing this wholesale from the Stan users group. Michael Betancourt provided this really good discussion of identifiability in Bayesian inference, which I believe bears on your request for a contrast of the two statistical schools.

The first difference with a Bayesian analysis will be the presence of priors which, even when weak, will constrain the posterior mass for those 4 parameters into a finite neighborhood (otherwise you wouldn't have had a valid prior in the first place). Despite this, you can still have non-identifiability in the sense that the posterior will not converge to a point mass in the limit of infinite data. In a very real sense, however, that doesn't matter because (a) the infinite data limit isn't real anyways and (b) Bayesian inference doesn't report point estimates but rather distributions. In practice such non-identifiability will result in large correlations between the parameters (perhaps even non-convexity) but a proper Bayesian analysis will identify those correlations. Even if you report single parameter marginals you'll get distributions that span the marginal variance rather than the conditional variance at any point (which is what a standard frequentist result would quote, and why identifiability is really important there), and it's really the marginal variance that best encodes the uncertainty regarding a parameter.

Simple example: consider a model with parameters $\mu_1$ and $\mu_2$ with likelihood $\mathcal{N}(x | \mu_1 + \mu_2, \sigma)$. No matter how much data you collect, the likelihood will not converge to a point but rather a line $\mu_1 + \mu_2 = 0$. The conditional variance of $\mu_1$ and $\mu_2$ at any point on that line will be really small, despite the fact that the parameters can't really be identified.

Bayesian priors constrain the posterior distribution from that line to a long, cigar shaped distribution. Not easily to sample from but at least compact. A good Bayesian analysis will explore the entirety of that cigar, either identifying the correlation between $\mu_1$ and $\mu_2$ or returning the marginal variances that correspond to the projection of the long cigar onto the $\mu_1$ or $\mu_2$ axes, which give a much more faithful summary of the uncertainty in the parameters than the conditional variances.

$\endgroup$
3
$\begingroup$

The key difference between Bayesian and frequentist approaches lies in the definition of a probability, so if it is necessary to treat probabilties strictly as a long run frequency then frequentist approaches are reasonable, if it isn't then you should use a Bayesian approach. If either interpretation is acceptable, then Bayesian and frequentist approaches are likely to be reasonable.

Another way of putting it, is if you want to know what inferences you can draw from a particular experiment, you probably want to be Bayesian; if you want to draw conclusions about some population of experiments (e.g. quality control) then frequentist methods are well suited.

Essentially, the important thing is to know what question you want answered, and choose the form of analysis that answers the question most directly.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.