Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

If the interest is merely estimating the parameters of a model (pointwise and/or interval estimation) and the prior information is not reliable, weak, (I know this is a bit vague but I am trying to establish an scenario where the choice of a prior is difficult) ... Why would someone choose to use the Bayesian approach with 'noninformative' improper priors instead of the classical approach?

share|improve this question
    
Thank you all for such interesting thoughts about this controversial part of the Bayesian statistics. I have been reading and comparing your points. There are interesting arguments validating its use in terms of formal rules, practicality and interpretation. I will select an answer at some point, but I am affraid this is going to be a very difficult task. –  user10525 May 3 '12 at 13:41
add comment

6 Answers

up vote 15 down vote accepted

Two reasons one may go with a Bayesian approach even if you're using highly non-informative priors:

  • Convergence problems. There are some distributions (binomial, negative binomial and generalized gamma are the ones I'm most familiar with) that have convergence issues a non-trivial amount of the time. You can use a "Bayesian" framework - and particular Markov chain Monte Carlo (MCMC) methods, to essentially plow through these convergence issues with computational power and get decent estimates from them.
  • Interpretation. A Bayesian estimate + 95% credible interval has a more intuitive interpretation than a frequentist estimate + 95% confidence interval, so some may prefer to simply report those.
share|improve this answer
    
MCMC is not really Bayesian method. You could simply draw estimates from your target likelihood (not posterior) if convergence is the issue. –  scottyaz Jul 17 '13 at 11:18
add comment

Although the results are going to be very similar, their interpretations differ.

Confidence intervals imply the notion of repeating an experiment many times and being able to capture the true parameter 95% of times. But you cannot say you have a 95% chance of capturing it.

Credible intervals (Bayesian), on the other hand, allow you to say that there is a 95% chance that the interval captures the true value.

This is just because you went from $P(Data|Hypothesis)$ to $P(Hypothesis|Data)$ using Baye's Rule.

share|improve this answer
1  
I may be confused here, but how does "the true value" fit into a Bayesian framework? Maybe you are referring to the posterior mode (or mean, or.. etc)? –  Macro May 4 '12 at 12:55
    
I'm referring to whatever parameter (population value) you're estimating with you're sample statistic, be it a mean, a mean difference, a regression slope... In brief, what you're after. –  dominic999 May 4 '12 at 18:42
1  
Yes, but doesn't "true value" indicate that the parameter is a constant (i.e. its distribution is a point mass)? The entire concept of looking at the posterior distribution seems to disagree with thinking of parameters in that way. –  Macro May 4 '12 at 18:44
    
Being no Bayesian expert, I'm sorry I can't argue here. Maybe you're right and my interpretation is possibly too naive (based on the notions I grasped here and there). –  dominic999 May 4 '12 at 21:18
add comment

Sir Harold Jeffreys was a strong proponent of the Bayesian approach. He showed that if you use diffuse improper priors the resulting Bayesian inference would be the same as the frequentist inferential approach (that is, Bayesian credible regions are the same as frequentist confidence intervals). Most Bayesians advocate proper informative priors. There are problems with improper priors and some can argue that no prior is truly non-informative. I think that the Bayesians that use these Jeffreys' prior do it as followers of Jeffreys. Dennis Lindley, one of the strongest advocates of the Bayesian approach, had a great deal of respect for Jeffreys but advocated informative priors.

share|improve this answer
    
+1 for the first few lines of your answer. In my opinion, the reason to choose a Jeffreys' prior over a "non-informative" prior is not simply as a follower of Jeffreys. It's because it really is like making no assumption whereas a so-called non-informative prior is making an assumption about the parametrization. –  Neil G May 3 '12 at 1:45
    
@NeilG I've also found some people like using them to essentially "Fail Frequentist" (in the same sense as Fail Safe) when using non-informative priors such that they can be interpreted by a naive reader. –  Fomite May 3 '12 at 1:47
    
@EpiGrad: What do you mean? (I'm sorry, my understanding of frequentist statistics is very poor.) –  Neil G May 3 '12 at 1:50
    
@NeilG Essentially exploiting that a Jeffrey's prior will give you what someone trained in frequentist fields is expecting to see. It's a decent middle ground when working in placed Bayesian methods haven't penetrated much. –  Fomite May 3 '12 at 2:01
    
@EpiGrad Interesting, thanks. –  Neil G May 3 '12 at 2:08
show 1 more comment

We could argue forever about foundations of inference to defend both approaches, but let me propose something different. A $\textit{practical}$ reason to favor a Bayesian analysis over a classical one is shown clearly by how both approaches deal with prediction. Suppose that we have the usual conditionally i.i.d. case. Classically, a predictive density is defined plugging the value $\hat{\theta} = \hat{\theta}(x_1,\dots,x_n)$ of an estimate of the parameter $\Theta$ into the conditional density $f_{X_{n+1}\mid\Theta}(x_{n+1}\mid\theta)$. This classical predictive density $f_{X_{n+1}\mid\Theta}(x_{n+1}\mid\hat{\theta})$ does not account for the uncertainty of the estimate $\hat{\theta}$: two equal point estimates with totally different confidence intervals give you the same predictive density. On the other hand, the Bayesian predictive density takes into account the uncertainty about the parameter, given the information in a sample of observations, automatically, since $$ f_{X_{n+1}\mid X_1,\dots,X_m}(x_{n+1}\mid x_1,\dots,x_n) = \int f_{X_{n+1}\mid\Theta}(x_{n+1}\mid\theta) \, \pi(\theta\mid x_1,\dots,x_n) \, d\theta \, . $$

share|improve this answer
4  
It's worth pointing out that in the context of linear regression with normal errors, the frequentist prediction intervals are based on pivotal statistics rather than plug-in estimators and are identical to the Bayesian intervals under the typical noninformative priors (jointly flat on the $\beta$s and $\mathrm{log}(\sigma^2)$). –  Cyan May 3 '12 at 4:51
    
Related to @Cyan's comment. –  user10525 Oct 13 '12 at 18:35
add comment

The Bayesian approach has practical advantages. It helps with estimation, often being mandatory. And it enables novel model families, and helps in construction of more complicated (hierarchical, multilevel) models.

For example, with mixed models (including random effects with variance parameters) one gets better estimates if variance parameters are estimated by marginalizing over lower-level parameters (model coefficients; this is called REML). The Bayesian approach does this naturally. With these models, even with REML, maximum likelyhood (ML) estimates of variance parameters are often zero, or downward biased. A proper prior for the variance parameters helps.

Even if point estimation (MAP, maximum a posteriori) is used, priors change the model family. Linear regression with a large set of somewhat collinear variables is unstable. L2 regularization is used as a remedy, but it is interpretable as a Bayesian model with Gaussian (non-informative) prior, and MAP estimation. (L1 regularization is a different prior and gives different results. Actually here the prior may be somewhat informative, but it is about the collective properties of the parameters, not about a single parameter.)

So there are some common and relatively simple models where a Bayesian approach is needed just to get the thing done!

Things are even more in favor with more complicated models, such as the latent Dirichlet allocation (LDA) used in machine learning. And some models are inherently Bayesian, e.g., those based on Dirichlet processes.

share|improve this answer
add comment

I believe one reason to do so is that a Bayesian analysis provides you with a full posterior distribution. This can result in more detailed intervals than the typical frequentist $\pm 2 \sigma$. An applicable quote, from Reis and Stedinger 2005, is:

Providing a full posterior distribution of the parameters is an advantage of the Bayesian approach over classical methods, which usually provide only a point estimate of the parameters represented by the mode of the likelihood function, and make use of asymptotic normality assumptions and a quadratic approximation of the log-likelihood function to describe uncertainties. With the Bayesian framework, one does not have to use any approximation to evaluate the uncertainties because the full posterior distribution of the parameters is available. Moreover, a Bayesian analysis can provide credible intervals for parameters or any function of the parameters which are more easily interpreted than the concept of confidence interval in classical statistics (Congdon, 2001).

So, for example, you can calculate credible intervals for the difference between two parameters.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.