44
$\begingroup$

What are some practical objections to the use of Bayesian statistical methods in any context? No, I don't mean the usual carping about choice of prior. I'll be delighted if this gets no answers.

$\endgroup$
  • 5
    $\begingroup$ The question is framed ok, but the comments are getting right up to the line of argumentation and threatening to spill over to the wrong side of that line. Be careful...this is not the place for such debate. Create a chat room if you want to do that. $\endgroup$ – whuber Oct 18 '11 at 2:50
40
$\begingroup$

I'm going to give you an answer. Four drawbacks actually. Note that none of these are actually objections that should drive one all the way to frequentist analysis, but there are cons to going with a Bayesian framework:

  1. Choice of prior. This is the usual carping for a reason, though in my case it's not the usual "priors are subjective!" but that coming up with a prior that's well reasoned and actually represents your best attempt at summarizing a prior is a great deal of work in many cases. An entire aim of my dissertation, for example, can be summed up as "estimate priors".
  2. It's computationally intensive. Especially for models involving many variables. For a large dataset with many variables being estimated, it may very well be prohibitively computationally intensive, especially in certain circumstances where the data cannot readily be thrown onto a cluster or the like. Some of the ways to resolve this, like augmented data rather than MCMC, are somewhat theoretically challenging, at least to me.
  3. Posterior distributions are somewhat more difficult to incorporate into a meta-analysis, unless a frequentist, parametric description of the distribution has been provided.
  4. Depending on what journal the analysis is intended for, either the use of Bayes generally, or your choice of priors, gives your paper slightly more points where a reviewer can dig into it. Some of these are reasonable reviewer objections, but some just stem from the nature of Bayes and how familiar people in some fields are with it.

None of these things should stop you. Indeed, none of these things have stopped me, and hopefully doing Bayesian analysis will help address at least number 4.

$\endgroup$
  • 2
    $\begingroup$ #1, this, ideally, should be first stage analysis. In arts a lit review. In sciences a quantitative lit review. Bayesians shouldn't be apologetic about it. IF freqs approach data as if they're Adam and Eve - fine. The 1st chapter of my PhD is a meta-analysis (albeit frequentist).Whoopdeedoo. That's the way it should be. #2 Moore's law, I've found a brief and XKCD-based discussion with the local High Performance Computing group can help a lot. #3 Meta Analysis sucks either way. I would be in favour of mandatory rolling mega-analysis, in other words - provide your data when you publish. $\endgroup$ – rosser Oct 17 '11 at 21:49
  • 7
    $\begingroup$ @rosser A few thoughts. #1. There should indeed be a lit review, and yes, that should be step one. But a proper Bayesian analysis that controls for confounding properly needs a full, quantitative lit review of every variable to be included in the model. That's no small task. #2. Depending on Moore's law is a bad idea. First, recent gains have been made mostly in multi-core/GPU systems. That needs software written for it, and problems that gain from parallel processing. A single GLM model done with MCMC might not be that. Cont... $\endgroup$ – Fomite Oct 17 '11 at 21:53
  • 6
    $\begingroup$ @rosser and there may be times with HPC isn't necessarily the answer. For example, I work in areas where data use agreements and the like often prevent data from being stored on things besides extremely secure systems. The local cluster...isn't that. And in the end, Moore's Law is only as good as your hardware budget is large. As for #3 and meta-analysis, I tend to disagree, but beyond that, it remains a problem up until the point an entirely open-data system becomes the norm. $\endgroup$ – Fomite Oct 17 '11 at 21:55
  • $\begingroup$ OK I overstated #3. But how much difference does your prior on EVERY PREDICTOR make to the outcome? srsly? Does a sensitivity analysis show enormous differences? $\endgroup$ – rosser Oct 17 '11 at 22:14
  • 1
    $\begingroup$ @Rosser It likely depends on the nature of your predictor, and its relationship to the exposure and outcome. But in order to do a sensitivity analysis someone has to have a prior for all those variables. Perhaps I'll add it as a side-bit of my dissertation. I also find co-opting the strength of Bayes but assuming uninformative priors on variables where "I can't be bothered to find out" somewhat problematic. $\endgroup$ – Fomite Oct 17 '11 at 22:19
16
$\begingroup$

I am a Bayesian by inclination, but generally a frequentist in practice. The reason for this is usually that performing the full Bayesian analysis properly (rather than e.g. MAP solutions) for the types of problem I am interested in is tricky and computationally intensive. Often a full Bayesian analysis is required to really see the benefit of this approach over frequentist equivalents.

For me, the trade-off is basically a choice between Bayesian methods that are conceptually elegant and easy to understand, but difficult to implement in practice and frequentist methods, which are conceptually awkward and subtle (try explaining how to interpret a hypothesis test accurately or why there isn't a 95% probability that the true value lies in a 95% confidence interval), but which are well suited to easily implemented "cookbook" solutions.

Horses for courses.

$\endgroup$
11
$\begingroup$

From a purely practical point of view, I am not a fan of methods which require lots of computation (I am thinking of Gibbs sampler and MCMC, often used in the Bayesian framework, but this also applies to e.g. bootstrap techniques in frequentist analysis). The reason being that any kind of debugging (testing the implementation, looking at robustness with respect to assumptions, etc) itself requires a bunch of Monte Carlo simulations, and you are quickly in a computational morass. I prefer the underlying analysis techniques to be fast and deterministic, even if they are only approximate.

This is a purely practical objection, of course: given infinite computing resources, this objection would disappear. And it only applies to a subset of Bayesian methods. Also this is more of a preference given my workflow.

$\endgroup$
  • 1
    $\begingroup$ So far I'm hearing 1. Moore'e Law, 2. Hard work +/- patience and 3. Ignorance. Have to say none of these are convincing. Bayes seems like such an over-arching paradigm. For example ... why weren't GWAS studies analysed a-la Bayes. Could they have prevented throwing 99.999% of the data away? $\endgroup$ – rosser Oct 17 '11 at 21:36
  • 1
    $\begingroup$ Conversely: MCMC could teach one to write faster code and to learn from the pain of waiting for simulations to complete. This has been my experience with modeling: if it takes a long time to run, I may benefit from learning how to make the code faster. $\endgroup$ – Iterator Oct 18 '11 at 19:52
9
$\begingroup$

Sometimes there's a simple and natural "classical" solution to a problem, in which case a fancy Bayesian method (especially with MCMC) would be overkill.

Further, in variable selection type problems, it can be more straightforward and clear to consider something like a penalized likelihood; there may exist a prior on models that gives an equivalent Bayesian approach, but how the prior corresponds to the ultimate performance can be less clear than the relationship between the penalty and performance.

Finally, MCMC methods often require an expert both for assessing convergence/mixing and for making sense of the results.

$\endgroup$
9
$\begingroup$

I am relatively new to Bayesian methods, but one thing that that irks me is that, while I understand the rationale of priors (i.e. science is a cumulative endeavour, so for most questions there is some amount of previous experience/thinking that should inform your interpretation of the data), I dislike that the Bayesian approach forces you to push subjectivity to the beginning of the analysis, rendering the end result contingent. I believe this is problematic for two reasons: 1) some less well versed readers won't even pay attention to the priors, and interpret Bayesian results as non-contingent; 2) unless the raw data is available, it is hard for readers to reframe the results in their own subjective priors. This is why I prefer likelihood ratios, they put the subjectivity at the end by simply quantifying relative evidence values and leaving it up to the reader to apply their own subjective criteria to determine whether they believe the evidence is sufficiently in favor of either model involved in the ratio.

(Astute critics will note that even the likelihood ratio is "contingent" in the sense that it is contingent on the parameterization of the models being compared; however this is a feature shared by all methods, Frequentist, Bayesian and Likelihoodist)

$\endgroup$
  • 9
    $\begingroup$ The converse problem with frequentist statstics is that the subjectivity is there, but it isn't mentioned at all. The (practical) trouble with likelihood ratios is that they are based on optimising the likelihood and hence ignore the fact that there may be other solutions with a likelihood only slightly less. That is where the Bayes factor is useful. But it is always "horses for courses". $\endgroup$ – Dikran Marsupial Oct 18 '11 at 12:01
6
$\begingroup$

Decision theory is the underlying theory on which statistics operates. The problem is to find a good (in some sense) procedure for producing decisions from data. However, there's rarely an unambiguous choice of procedure, in the sense of minimizing expected loss, so other criteria must be invoked to choose among them. Choosing the procedures that is Bayes with respect to some prior is one of these criteria, but it may not always be what you want. Minimax might be more important in some case, or unbiasedness.

Anyone who insists that the frequentists are wrong or the Bayesians or wrong is mostly revealing their ignorance of statistics.

$\endgroup$
5
$\begingroup$

For some time I have wanted to educate myself more on Bayesian approaches to modeling to get past my cursory understanding (I have coded Gibbs samplers in graduate course work, but have never done anything real). Along the way though I have thought some of Brian Dennis' papers have been though-provoking and have wished I could find a Bayesian friend (the ones who weren't in the closet) to read the papers and hear their counterpoints. So, here are the papers I am referring to, but the quote I always remember is

Being Bayesian means never having to say you're wrong.

http://faculty.washington.edu/skalski/classes/QERM597/papers/Dennis_1996.pdf http://classes.warnercnr.colostate.edu/nr575/files/2011/01/Lele-and-Dennis-2009.pdf

$\endgroup$
  • 1
    $\begingroup$ The first paper (haven't read the second) seems to more about how bayes is practiced vs the theory. In practice models aren't checked as rigorously as they should, but in theory bayesian statistics has superior model check facilities, called the "evidence" by Jaynes, which is embodied in the denominator P(D|model) of bayes' rule. With it you can compare the appropriateness of a models, something you can only do empirically in frequentist stats. The problem, of course, is that the evidence is hard to compute, so most people ignore it and think the posterior is the all-important factor(cont'd) $\endgroup$ – cespinoza Oct 18 '11 at 0:48
  • 2
    $\begingroup$ pt. 2 Try googling "skilling nested sampling" and you'll find a paper on an MCMC method for computing the evidence. (There are other, non-evidence based model checking methods as well: Gelman checks his models by sampling from the posterior predictive and comparing that (visually or otherwise) to the actual data.) Some people even suggest that models should be averaged by viewing the space of models itself to marginalize over. Another thing we can see on the horizon is nonparametric bayes, which solves the issue by allowing a much wider range of models than traditional parametric models. $\endgroup$ – cespinoza Oct 18 '11 at 0:58
  • 5
    $\begingroup$ Also, I suggest you watch videolectures.net/mlss09uk_jordan_bfway by Michael I. Jordan, a prof at berkeley who is quite balanced in his views on the supposed Bayes vs Freq. "war". I can't really comment on the second half of the first paper b/c I don't know any of the ecological references. I'll read the second one later. $\endgroup$ – cespinoza Oct 18 '11 at 1:04
  • 1
    $\begingroup$ @cespinoza: I was thinking this on the way in to work. The paper says a Bayesian would never look at residuals (i.e. compare model output to actual data), and perhaps a strident Bayesian might eschew this on principle, but practitioners like Gelman certainly do compare model output (predictive posterior) to actual data. I don't know enough to go further, but my impression of the papers are that they set up "in principle" straw men to attack. $\endgroup$ – Wayne Oct 18 '11 at 13:09
  • 1
    $\begingroup$ Just add that a Bayesian who doesn't check residuals is a bad statistician. Usually, a Bayesian method is used with a "rough and ready" model and prior. Checking residuals is one way to see if you've got enough of your knowledge into the prior and the model. It goes hand in hand with checking what theoretical features your model and prior have $\endgroup$ – probabilityislogic Jan 24 '12 at 14:53
5
$\begingroup$

What are the open problems in Bayesian Statistics from the ISBA quarterly newsletter list 5 problems with bayesian stats from various leaders in the field, #1 being, boringly enough, "Model selection and hypothesis testing".

$\endgroup$
  • 3
    $\begingroup$ Why boringly?! This is one of the most frequent uses of statistical methods. $\endgroup$ – Xi'an Dec 3 '11 at 12:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.