Why would someone use a Bayesian approach with a 'noninformative' improper prior instead of the classical approach?

Question

If the interest is merely estimating the parameters of a model (pointwise and/or interval estimation) and the prior information is not reliable, weak, (I know this is a bit vague but I am trying to establish an scenario where the choice of a prior is difficult) ... Why would someone choose to use the Bayesian approach with 'noninformative' improper priors instead of the classical approach?

Answer 1

Two reasons one may go with a Bayesian approach even if you're using highly non-informative priors:

• Convergence problems. There are some distributions (binomial, negative binomial and generalized gamma are the ones I'm most familiar with) that have convergence issues a non-trivial amount of the time. You can use a "Bayesian" framework - and particular Markov chain Monte Carlo (MCMC) methods, to essentially plow through these convergence issues with computational power and get decent estimates from them.
• Interpretation. A Bayesian estimate + 95% credible interval has a more intuitive interpretation than a frequentist estimate + 95% confidence interval, so some may prefer to simply report those.

Answer 2

Although the results are going to be very similar, their interpretations differ.

Confidence intervals imply the notion of repeating an experiment many times and being able to capture the true parameter 95% of times. But you cannot say you have a 95% chance of capturing it.

Credible intervals (Bayesian), on the other hand, allow you to say that there is a 95% chance that the interval captures the true value.

This is just because you went from $P(Data|Hypothesis)$ to $P(Hypothesis|Data)$ using Baye's Rule.

Answer 3

Sir Harold Jeffreys was a strong proponent of the Bayesian approach. He showed that if you use diffuse improper priors the resulting Bayesian inference would be the same as the frequentist inferential approach (that is, Bayesian credible regions are the same as frequentist confidence intervals). Most Bayesians advocate proper informative priors. There are problems with improper priors and some can argue that no prior is truly non-informative. I think that the Bayesians that use these Jeffreys' prior do it as followers of Jeffreys. Dennis Lindley, one of the strongest advocates of the Bayesian approach, had a great deal of respect for Jeffreys but advocated informative priors.

Answer 4

We could argue forever about foundations of inference to defend both approaches, but let me propose something different. A $\textit{practical}$ reason to favor a Bayesian analysis over a classical one is shown clearly by how both approaches deal with prediction. Suppose that we have the usual conditionally i.i.d. case. Classically, a predictive density is defined plugging the value $\hat{\theta} = \hat{\theta}(x_1,\dots,x_n)$ of an estimate of the parameter $\Theta$ into the conditional density $f_{X_{n+1}\mid\Theta}(x_{n+1}\mid\theta)$. This classical predictive density $f_{X_{n+1}\mid\Theta}(x_{n+1}\mid\hat{\theta})$ does not account for the uncertainty of the estimate $\hat{\theta}$: two equal point estimates with totally different confidence intervals give you the same predictive density. On the other hand, the Bayesian predictive density takes into account the uncertainty about the parameter, given the information in a sample of observations, automatically, since $$ f_{X_{n+1}\mid X_1,\dots,X_m}(x_{n+1}\mid x_1,\dots,x_n) = \int f_{X_{n+1}\mid\Theta}(x_{n+1}\mid\theta) \, \pi(\theta\mid x_1,\dots,x_n) \, d\theta \, . $$

Answer 5

The Bayesian approach has practical advantages. It helps with estimation, often being mandatory. And it enables novel model families, and helps in construction of more complicated (hierarchical, multilevel) models.

For example, with mixed models (including random effects with variance parameters) one gets better estimates if variance parameters are estimated by marginalizing over lower-level parameters (model coefficients; this is called REML). The Bayesian approach does this naturally. With these models, even with REML, maximum likelyhood (ML) estimates of variance parameters are often zero, or downward biased. A proper prior for the variance parameters helps.

Even if point estimation (MAP, maximum a posteriori) is used, priors change the model family. Linear regression with a large set of somewhat collinear variables is unstable. L2 regularization is used as a remedy, but it is interpretable as a Bayesian model with Gaussian (non-informative) prior, and MAP estimation. (L1 regularization is a different prior and gives different results. Actually here the prior may be somewhat informative, but it is about the collective properties of the parameters, not about a single parameter.)

So there are some common and relatively simple models where a Bayesian approach is needed just to get the thing done!

Things are even more in favor with more complicated models, such as the latent Dirichlet allocation (LDA) used in machine learning. And some models are inherently Bayesian, e.g., those based on Dirichlet processes.

Answer 6

I believe one reason to do so is that a Bayesian analysis provides you with a full posterior distribution. This can result in more detailed intervals than the typical frequentist $\pm 2 \sigma$. An applicable quote, from Reis and Stedinger 2005, is:

Providing a full posterior distribution of the parameters is an advantage of the Bayesian approach over classical methods, which usually provide only a point estimate of the parameters represented by the mode of the likelihood function, and make use of asymptotic normality assumptions and a quadratic approximation of the log-likelihood function to describe uncertainties. With the Bayesian framework, one does not have to use any approximation to evaluate the uncertainties because the full posterior distribution of the parameters is available. Moreover, a Bayesian analysis can provide credible intervals for parameters or any function of the parameters which are more easily interpreted than the concept of confidence interval in classical statistics (Congdon, 2001).

So, for example, you can calculate credible intervals for the difference between two parameters.