44
$\begingroup$

If the interest is merely estimating the parameters of a model (pointwise and/or interval estimation) and the prior information is not reliable, weak, (I know this is a bit vague but I am trying to establish an scenario where the choice of a prior is difficult) ... Why would someone choose to use the Bayesian approach with 'noninformative' improper priors instead of the classical approach?

$\endgroup$
  • 1
    $\begingroup$ Thank you all for such interesting thoughts about this controversial part of the Bayesian statistics. I have been reading and comparing your points. There are interesting arguments validating its use in terms of formal rules, practicality and interpretation. I will select an answer at some point, but I am affraid this is going to be a very difficult task. $\endgroup$ – user10525 May 3 '12 at 13:41
24
$\begingroup$

Two reasons one may go with a Bayesian approach even if you're using highly non-informative priors:

  • Convergence problems. There are some distributions (binomial, negative binomial and generalized gamma are the ones I'm most familiar with) that have convergence issues a non-trivial amount of the time. You can use a "Bayesian" framework - and particular Markov chain Monte Carlo (MCMC) methods, to essentially plow through these convergence issues with computational power and get decent estimates from them.
  • Interpretation. A Bayesian estimate + 95% credible interval has a more intuitive interpretation than a frequentist estimate + 95% confidence interval, so some may prefer to simply report those.
$\endgroup$
  • 3
    $\begingroup$ MCMC is not really Bayesian method. You could simply draw estimates from your target likelihood (not posterior) if convergence is the issue. $\endgroup$ – scottyaz Jul 17 '13 at 11:18
16
$\begingroup$

Although the results are going to be very similar, their interpretations differ.

Confidence intervals imply the notion of repeating an experiment many times and being able to capture the true parameter 95% of times. But you cannot say you have a 95% chance of capturing it.

Credible intervals (Bayesian), on the other hand, allow you to say that there is a 95% "chance" that the interval captures the true value. Update: A more Bayesian way of putting it would be that you could be 95% confident about your results.

This is just because you went from $P(Data|Hypothesis)$ to $P(Hypothesis|Data)$ using Baye's Rule.

$\endgroup$
  • 1
    $\begingroup$ I may be confused here, but how does "the true value" fit into a Bayesian framework? Maybe you are referring to the posterior mode (or mean, or.. etc)? $\endgroup$ – Macro May 4 '12 at 12:55
  • $\begingroup$ I'm referring to whatever parameter (population value) you're estimating with you're sample statistic, be it a mean, a mean difference, a regression slope... In brief, what you're after. $\endgroup$ – Dominic Comtois May 4 '12 at 18:42
  • 1
    $\begingroup$ Yes, but doesn't "true value" indicate that the parameter is a constant (i.e. its distribution is a point mass)? The entire concept of looking at the posterior distribution seems to disagree with thinking of parameters in that way. $\endgroup$ – Macro May 4 '12 at 18:44
9
$\begingroup$

I believe one reason to do so is that a Bayesian analysis provides you with a full posterior distribution. This can result in more detailed intervals than the typical frequentist $\pm 2 \sigma$. An applicable quote, from Reis and Stedinger 2005, is:

Providing a full posterior distribution of the parameters is an advantage of the Bayesian approach over classical methods, which usually provide only a point estimate of the parameters represented by the mode of the likelihood function, and make use of asymptotic normality assumptions and a quadratic approximation of the log-likelihood function to describe uncertainties. With the Bayesian framework, one does not have to use any approximation to evaluate the uncertainties because the full posterior distribution of the parameters is available. Moreover, a Bayesian analysis can provide credible intervals for parameters or any function of the parameters which are more easily interpreted than the concept of confidence interval in classical statistics (Congdon, 2001).

So, for example, you can calculate credible intervals for the difference between two parameters.

$\endgroup$
6
$\begingroup$

Sir Harold Jeffreys was a strong proponent of the Bayesian approach. He showed that if you use diffuse improper priors the resulting Bayesian inference would be the same as the frequentist inferential approach (that is, Bayesian credible regions are the same as frequentist confidence intervals). Most Bayesians advocate proper informative priors. There are problems with improper priors and some can argue that no prior is truly non-informative. I think that the Bayesians that use these Jeffreys' prior do it as followers of Jeffreys. Dennis Lindley, one of the strongest advocates of the Bayesian approach, had a great deal of respect for Jeffreys but advocated informative priors.

$\endgroup$
  • 1
    $\begingroup$ +1 for the first few lines of your answer. In my opinion, the reason to choose a Jeffreys' prior over a "non-informative" prior is not simply as a follower of Jeffreys. It's because it really is like making no assumption whereas a so-called non-informative prior is making an assumption about the parametrization. $\endgroup$ – Neil G May 3 '12 at 1:45
  • 1
    $\begingroup$ @NeilG I've also found some people like using them to essentially "Fail Frequentist" (in the same sense as Fail Safe) when using non-informative priors such that they can be interpreted by a naive reader. $\endgroup$ – Fomite May 3 '12 at 1:47
  • $\begingroup$ @EpiGrad: What do you mean? (I'm sorry, my understanding of frequentist statistics is very poor.) $\endgroup$ – Neil G May 3 '12 at 1:50
  • 1
    $\begingroup$ @NeilG Essentially exploiting that a Jeffrey's prior will give you what someone trained in frequentist fields is expecting to see. It's a decent middle ground when working in placed Bayesian methods haven't penetrated much. $\endgroup$ – Fomite May 3 '12 at 2:01
  • $\begingroup$ @NeilG I also forgot that, as in my answer, if you're using MCMC to conduct a frequentist analysis, skirting around convergence issues, then the Jeffrey's prior is also helpful. $\endgroup$ – Fomite May 3 '12 at 2:11
6
$\begingroup$

The Bayesian approach has practical advantages. It helps with estimation, often being mandatory. And it enables novel model families, and helps in construction of more complicated (hierarchical, multilevel) models.

For example, with mixed models (including random effects with variance parameters) one gets better estimates if variance parameters are estimated by marginalizing over lower-level parameters (model coefficients; this is called REML). The Bayesian approach does this naturally. With these models, even with REML, maximum likelyhood (ML) estimates of variance parameters are often zero, or downward biased. A proper prior for the variance parameters helps.

Even if point estimation (MAP, maximum a posteriori) is used, priors change the model family. Linear regression with a large set of somewhat collinear variables is unstable. L2 regularization is used as a remedy, but it is interpretable as a Bayesian model with Gaussian (non-informative) prior, and MAP estimation. (L1 regularization is a different prior and gives different results. Actually here the prior may be somewhat informative, but it is about the collective properties of the parameters, not about a single parameter.)

So there are some common and relatively simple models where a Bayesian approach is needed just to get the thing done!

Things are even more in favor with more complicated models, such as the latent Dirichlet allocation (LDA) used in machine learning. And some models are inherently Bayesian, e.g., those based on Dirichlet processes.

$\endgroup$
6
$\begingroup$

We could argue forever about foundations of inference to defend both approaches, but let me propose something different. A $\textit{practical}$ reason to favor a Bayesian analysis over a classical one is shown clearly by how both approaches deal with prediction. Suppose that we have the usual conditionally i.i.d. case. Classically, a predictive density is defined plugging the value $\hat{\theta} = \hat{\theta}(x_1,\dots,x_n)$ of an estimate of the parameter $\Theta$ into the conditional density $f_{X_{n+1}\mid\Theta}(x_{n+1}\mid\theta)$. This classical predictive density $f_{X_{n+1}\mid\Theta}(x_{n+1}\mid\hat{\theta})$ does not account for the uncertainty of the estimate $\hat{\theta}$: two equal point estimates with totally different confidence intervals give you the same predictive density. On the other hand, the Bayesian predictive density takes into account the uncertainty about the parameter, given the information in a sample of observations, automatically, since $$ f_{X_{n+1}\mid X_1,\dots,X_m}(x_{n+1}\mid x_1,\dots,x_n) = \int f_{X_{n+1}\mid\Theta}(x_{n+1}\mid\theta) \, \pi(\theta\mid x_1,\dots,x_n) \, d\theta \, . $$

$\endgroup$
  • 6
    $\begingroup$ It's worth pointing out that in the context of linear regression with normal errors, the frequentist prediction intervals are based on pivotal statistics rather than plug-in estimators and are identical to the Bayesian intervals under the typical noninformative priors (jointly flat on the $\beta$s and $\mathrm{log}(\sigma^2)$). $\endgroup$ – Cyan May 3 '12 at 4:51
  • $\begingroup$ Related to @Cyan's comment. $\endgroup$ – user10525 Oct 13 '12 at 18:35
4
$\begingroup$

There are several reasons:

  1. In many situations constructing test statistics or confidence intervals is quite difficult, because normal approximations – even after using an appropriate link function – to work with $\pm \text{SE}$ are often not working too well for sparse data situations. By using Bayesian inference with uninformative priors implemented via MCMC you get around this (for caveats see below).
  2. The large sample properties are usually completely identical to some corresponding frequentist approach.
  3. There is often considerable reluctance to agree on any priors, no matter how much we actually know, due to a fear of being accused of “not being objective”. By using uninformative priors (“no priors”) one can pretend that there is no such issue, which will avoid criticism from some reviewers.

Now as to the downsides of just using uninformative priors, starting with what I think is the most important and then heading for some of the also quite important technical aspects:

  1. The interpretation of what you get is, quite honestly, much the same as for frequentist inference. You cannot just re-label your frequentist maximum likelihood inference as Bayesian maximum a-posteriori inference and claim that this absolves you of any worries about multiple comparisons, multiple looks at the data and lets you interpret all statements in terms of the probability that some hypothesis is true. Sure, type I errors and so on are frequentist concepts, but we should as scientists care about making false claims and we know that doing the above causes problems. A lot of these issues go away (or at least are a lot less of a problem), if you embed things in a hierarchical model / do something empirical Bayes, but that usually boils down to implicitly generating priors via the analysis procedure by including the basis for your prior in your model (and an alternative to that is to explicitly formulate priors). These considerations are frequently ignored, in my opinion mostly to conduct Bayesian p-hacking (i.e. introduce multiplicity, but ignore it) with the fig-leaf of an excuse that this is no problem when you use Bayesian methods (omitting all the conditions that would have to be fulfilled).
  2. On the more “technical” side, uninformative priors are problematic, because you are not guaranteed a proper posterior. Many people have fitted Bayesian models with uninformative priors and not realized that the posterior is not proper. As a result MCMC samples were generated that were essentially meaningless.

The last point is an argument for preferring rather vague (or slightly more weakly-informative) priors that ensure a proper posterior. Admittedly, it can sometimes be hard to sample from these, too, and it may be hard to notice that the whole posterior has not been explored. However, Bayesian methods with vague (but proper) priors have in many fields been shown to have really good small sample properties from a frequentist perspective and you could certainly see that as an argument for using those, while with somewhat more data there will be hardly any difference versus methods with uninformative priors.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy