Why would someone use a Bayesian approach with a 'noninformative' improper prior instead of the classical approach?

Question

If the interest is merely estimating the parameters of a model (pointwise and/or interval estimation) and the prior information is not reliable, weak, (I know this is a bit vague but I am trying to establish an scenario where the choice of a prior is difficult) ... Why would someone choose to use the Bayesian approach with 'noninformative' improper priors instead of the classical approach?

Answer 1

Two reasons one may go with a Bayesian approach even if you're using highly non-informative priors:

• Convergence problems. There are some distributions (binomial, negative binomial and generalized gamma are the ones I'm most familiar with) that have convergence issues a non-trivial amount of the time. You can use a "Bayesian" framework - and particular Markov chain Monte Carlo (MCMC) methods, to essentially plow through these convergence issues with computational power and get decent estimates from them.
• Interpretation. A Bayesian estimate + 95% credible interval has a more intuitive interpretation than a frequentist estimate + 95% confidence interval, so some may prefer to simply report those.

Answer 2

Although the results are going to be very similar, their interpretations differ.

Confidence intervals imply the notion of repeating an experiment many times and being able to capture the true parameter 95% of times. But you cannot say you have a 95% chance of capturing it.

Credible intervals (Bayesian), on the other hand, allow you to say that there is a 95% "chance" that the interval captures the true value. Update: A more Bayesian way of putting it would be that you could be 95% confident about your results.

This is just because you went from $P(Data|Hypothesis)$ to $P(Hypothesis|Data)$ using Baye's Rule.

Answer 3

I believe one reason to do so is that a Bayesian analysis provides you with a full posterior distribution. This can result in more detailed intervals than the typical frequentist $\pm 2 \sigma$. An applicable quote, from Reis and Stedinger 2005, is:

Providing a full posterior distribution of the parameters is an advantage of the Bayesian approach over classical methods, which usually provide only a point estimate of the parameters represented by the mode of the likelihood function, and make use of asymptotic normality assumptions and a quadratic approximation of the log-likelihood function to describe uncertainties. With the Bayesian framework, one does not have to use any approximation to evaluate the uncertainties because the full posterior distribution of the parameters is available. Moreover, a Bayesian analysis can provide credible intervals for parameters or any function of the parameters which are more easily interpreted than the concept of confidence interval in classical statistics (Congdon, 2001).

So, for example, you can calculate credible intervals for the difference between two parameters.

Answer 4

Sir Harold Jeffreys was a strong proponent of the Bayesian approach. He showed that if you use diffuse improper priors the resulting Bayesian inference would be the same as the frequentist inferential approach (that is, Bayesian credible regions are the same as frequentist confidence intervals). Most Bayesians advocate proper informative priors. There are problems with improper priors and some can argue that no prior is truly non-informative. I think that the Bayesians that use these Jeffreys' prior do it as followers of Jeffreys. Dennis Lindley, one of the strongest advocates of the Bayesian approach, had a great deal of respect for Jeffreys but advocated informative priors.

Answer 5

We could argue forever about foundations of inference to defend both approaches, but let me propose something different. A $\textit{practical}$ reason to favor a Bayesian analysis over a classical one is shown clearly by how both approaches deal with prediction. Suppose that we have the usual conditionally i.i.d. case. Classically, a predictive density is defined plugging the value $\hat{\theta} = \hat{\theta}(x_1,\dots,x_n)$ of an estimate of the parameter $\Theta$ into the conditional density $f_{X_{n+1}\mid\Theta}(x_{n+1}\mid\theta)$. This classical predictive density $f_{X_{n+1}\mid\Theta}(x_{n+1}\mid\hat{\theta})$ does not account for the uncertainty of the estimate $\hat{\theta}$: two equal point estimates with totally different confidence intervals give you the same predictive density. On the other hand, the Bayesian predictive density takes into account the uncertainty about the parameter, given the information in a sample of observations, automatically, since $$ f_{X_{n+1}\mid X_1,\dots,X_m}(x_{n+1}\mid x_1,\dots,x_n) = \int f_{X_{n+1}\mid\Theta}(x_{n+1}\mid\theta) \, \pi(\theta\mid x_1,\dots,x_n) \, d\theta \, . $$

Answer 6

The Bayesian approach has practical advantages. It helps with estimation, often being mandatory. And it enables novel model families, and helps in construction of more complicated (hierarchical, multilevel) models.

For example, with mixed models (including random effects with variance parameters) one gets better estimates if variance parameters are estimated by marginalizing over lower-level parameters (model coefficients; this is called REML). The Bayesian approach does this naturally. With these models, even with REML, maximum likelyhood (ML) estimates of variance parameters are often zero, or downward biased. A proper prior for the variance parameters helps.

Even if point estimation (MAP, maximum a posteriori) is used, priors change the model family. Linear regression with a large set of somewhat collinear variables is unstable. L2 regularization is used as a remedy, but it is interpretable as a Bayesian model with Gaussian (non-informative) prior, and MAP estimation. (L1 regularization is a different prior and gives different results. Actually here the prior may be somewhat informative, but it is about the collective properties of the parameters, not about a single parameter.)

So there are some common and relatively simple models where a Bayesian approach is needed just to get the thing done!

Things are even more in favor with more complicated models, such as the latent Dirichlet allocation (LDA) used in machine learning. And some models are inherently Bayesian, e.g., those based on Dirichlet processes.

Answer 7

There are several reasons:

1. In many situations constructing test statistics or confidence intervals is quite difficult, because normal approximations – even after using an appropriate link function – to work with $\pm \text{SE}$ are often not working too well for sparse data situations. By using Bayesian inference with uninformative priors implemented via MCMC you get around this (for caveats see below).
2. The large sample properties are usually completely identical to some corresponding frequentist approach.
3. There is often considerable reluctance to agree on any priors, no matter how much we actually know, due to a fear of being accused of “not being objective”. By using uninformative priors (“no priors”) one can pretend that there is no such issue, which will avoid criticism from some reviewers.

Now as to the downsides of just using uninformative priors, starting with what I think is the most important and then heading for some of the also quite important technical aspects:

1. The interpretation of what you get is, quite honestly, much the same as for frequentist inference. You cannot just re-label your frequentist maximum likelihood inference as Bayesian maximum a-posteriori inference and claim that this absolves you of any worries about multiple comparisons, multiple looks at the data and lets you interpret all statements in terms of the probability that some hypothesis is true. Sure, type I errors and so on are frequentist concepts, but we should as scientists care about making false claims and we know that doing the above causes problems. A lot of these issues go away (or at least are a lot less of a problem), if you embed things in a hierarchical model / do something empirical Bayes, but that usually boils down to implicitly generating priors via the analysis procedure by including the basis for your prior in your model (and an alternative to that is to explicitly formulate priors). These considerations are frequently ignored, in my opinion mostly to conduct Bayesian p-hacking (i.e. introduce multiplicity, but ignore it) with the fig-leaf of an excuse that this is no problem when you use Bayesian methods (omitting all the conditions that would have to be fulfilled).
2. On the more “technical” side, uninformative priors are problematic, because you are not guaranteed a proper posterior. Many people have fitted Bayesian models with uninformative priors and not realized that the posterior is not proper. As a result MCMC samples were generated that were essentially meaningless.

The last point is an argument for preferring rather vague (or slightly more weakly-informative) priors that ensure a proper posterior. Admittedly, it can sometimes be hard to sample from these, too, and it may be hard to notice that the whole posterior has not been explored. However, Bayesian methods with vague (but proper) priors have in many fields been shown to have really good small sample properties from a frequentist perspective and you could certainly see that as an argument for using those, while with somewhat more data there will be hardly any difference versus methods with uninformative priors.