20
$\begingroup$

I have been learning about Bayesian statistics, and I often have read in articles

"we adopt a Bayesian approach"

or something similar. I also noticed, less often:

"we adopt a fully Bayesian approach"

(my emphasis). Is there any difference between these approaches in any practical or theoretical sense ? FWIW, I am using the package MCMCglmm in R in case that is relevant.

$\endgroup$
  • 6
    $\begingroup$ I don't think that "fully Bayesian" has a rigorous meaning. $\endgroup$ – Stéphane Laurent Jul 7 '12 at 20:38
  • 4
    $\begingroup$ @Stephane I am pretty sure that fully Bayesian is the same as Bayesian but the adjective fully is used to emphasize that it is not empirical Bayes. $\endgroup$ – Michael R. Chernick Jul 7 '12 at 21:38
  • 1
    $\begingroup$ @Michael this makes sense but I still think the meaning is not universal, and it seems to be confirmed by the several different answers to the question. I would not be surprised that some people say "fully bayesian" to say that they use a subjective prior and not a noninformative one. Another possible situation is when people use the "Bayesian-frequentist predictive distribution" and then move to a purely Bayesian approach. $\endgroup$ – Stéphane Laurent Jul 8 '12 at 9:01
  • $\begingroup$ @Stephane i accept your judgement. I think you work in Bayesian statistics more than I do and so have probably heard people use the term in various ways. At least my answer is snesible and partially right. $\endgroup$ – Michael R. Chernick Jul 8 '12 at 12:33
  • $\begingroup$ @MichaelChernick yes, your answer is an example of a pseudo-Bayesian approach vs a true Bayesian approach, but there are others such situations $\endgroup$ – Stéphane Laurent Jul 8 '12 at 12:48
19
+250
$\begingroup$

The terminology "fully Bayesian approach" is nothing but a way to indicate that one moves from a "partially" Bayesian approach to a "true" Bayesian approach, depending on the context. Or to distinguish a "pseudo-Bayesian" approach from a "strictly" Bayesian approach.

For example one author writes: "Unlike the majority of other authors interested who typically used an Empirical Bayes approach for RVM, we adopt a fully Bayesian approach" beacuse the empirical Bayes approach is a "pseudo-Bayesian" approach. There are others pseudo-Bayesian approaches, such as the Bayesian-frequentist predictive distribution (a distribution whose quantiles match the bounds of the frequentist prediction intervals).

In this page several R packages for Bayesian inference are presented. The MCMCglmm is presented as a "fully Bayesian approach" because the user has to choose the prior distribution, contrary to the other packages.

Another possible meaning of "fully Bayesian" is when one performs a Bayesian inference derived from the Bayesian decision theory framework, that is, derived from a loss function, because Bayesian decision theory is a solid foundational framework for Bayesian inference.

$\endgroup$
  • $\begingroup$ Thank you for this. thank you, so the package MCMCglmm being "Fully Bayesian" has nothing to do with using MCMC to derive the estimates and it would it still be fully Bayesian if I have to specify the prior, from which the posterior could be found analytically ? I'm sorry if my question doesn't make sense - I'm still a beginner, but I'm trying to learn ! $\endgroup$ – Joe King Jul 8 '12 at 12:08
  • 1
    $\begingroup$ MCMC is just a technique which is useful for simulating the posterior distributions in Bayesian statistics. But it has nothing to do with the Bayesian approach itself. $\endgroup$ – Stéphane Laurent Jul 8 '12 at 12:46
13
$\begingroup$

I think the terminology is used to distinguish between the Bayesian approach and the empirical Bayes approach. Full Bayes uses a specified prior whereas empirical Bayes allows the prior to be estimated through use of data.

$\endgroup$
  • $\begingroup$ Thank you ! I have also seen "empirical Bayes" mentioned here and there, but it never cropped up in things I've read, to the point where I had to think seriously about what it means. I just looked at the wikipedia page which says it is also known as "maximum marginal likelihood", and "an approximation to a fully Bayesian treatment of a hierarchical Bayes model". Hmmm, to be honest I don't understand very much of what is on that page :( $\endgroup$ – Joe King Jul 7 '12 at 17:22
  • $\begingroup$ @JoeKing There is a lot of interesting and important uses of empirical Bayes methods. The idea goes back to Herbert Robbins in the 1960s. In the 1970s Efron and Morris showed that the James-Stein estimator of a multivariate normal mean and other similar shrinkage estimators are empirical Bayes. In his new book on Large Scale Inference, Brad Efron shows how empirical Bayes methods can be used for problems sometimes called small n large p because many hypotheses are on parameters are tested with relatively small sample sizes (i.e. p can be a lot larger thaan n). This come up with microarrays. $\endgroup$ – Michael R. Chernick Jul 7 '12 at 17:40
  • 1
    $\begingroup$ Thank you again. I have to admit that I don't understand all of what you just wrote but I'm going to use it as my starting point for further study on this matter. $\endgroup$ – Joe King Jul 7 '12 at 17:56
9
$\begingroup$

"Bayesian" really means "approximate Bayesian".

"Fully Bayesian" also means "approximate Bayesian", but with less approximation.

Edit: Clarification.

The fully Bayesian approach would be, for a given model and data, to calculate the posterior probability using the Bayes rule $$ p(\theta \mid \text{Data}) \propto p(\text{Data} \mid \theta)p(\theta) \>.$$ Except for very simple models, this has too large computational complexity, and approximations are necessary. More accurate approximations, such as using MCMC with Gibbs sampling for all parameters $\theta$, are sometimes called "Fully Bayesian". Less accurate approximations, such as using point estimation for some parameters, cannot be called "Fully Bayesian". Some approximate inference methods are in-between, like Variational Bayes or Expectation Propagation, and they are sometimes (rarely) also called "Fully Bayesian".

$\endgroup$
  • $\begingroup$ Thank you. I read here that MCMCglmm package I am using is Fully Bayesian. Is that because it is using MCMC together with a prior for parameters ? $\endgroup$ – Joe King Jul 8 '12 at 7:25
  • $\begingroup$ @Arek I am really not convinced. So when I use a standard conjugate prior I am "more than fully" Bayesian ? And why do you claim that a point estimate is less "accurate" than posterior simulations ? $\endgroup$ – Stéphane Laurent Jul 8 '12 at 9:04
  • 1
    $\begingroup$ @StéphaneLaurent I do not claim that point estimation is always less accurate. Where are the yesterday's comments to my answer? $\endgroup$ – Arek Paterek Jul 8 '12 at 9:25
  • 1
    $\begingroup$ @ArekPaterek Your short answer looked like a joke and so those comments which don't apply to your revised answer does not apply to the revised one. So my guess is that a moderator probably removed them. Still calling fully Bayesian approximate is puzzling. $\endgroup$ – Michael R. Chernick Jul 8 '12 at 12:47
  • 1
    $\begingroup$ Maybe my first non-deleted comment was not clear. If Arek's answer were right, then how should we call the situation when it is possible to have the exact posterior distribution (such as a simple conjugate prior situation) ? A "more-than-fully" Bayesian approach ? $\endgroup$ – Stéphane Laurent Jul 8 '12 at 13:12
8
$\begingroup$

I would use "fully Bayesian" to mean that any nuissance parameters had been marginalised from the analysis, rather than optimised (e.g. MAP estimates). For example a Gaussian process model, with hyper-parameters tuned to maximise the marginal likelihood would be Bayesian, but only partially so, whereas if the hyper-parameters defining the covariance function were integrated out using a hyper-prior, that would be fully Bayesian.

$\endgroup$
  • 4
    $\begingroup$ This seems to be the slightly more general answer. The more quantities that are marginalised rather than optimised the more 'fully Bayesian' the solution is. Empirical Bayes is a special case. $\endgroup$ – conjugateprior Jul 7 '12 at 22:16
  • $\begingroup$ Yes, it is only a slight extension on Michaels' answer; essentially optimisation is fundamentally un-Bayesian. $\endgroup$ – Dikran Marsupial Jul 9 '12 at 10:34
3
$\begingroup$

As a practical example:

I do some Bayesian modeling using splines. A common problem with splines is knot selection. One popular possibility is to use a Reversible Jump Markov Chain Monte Carlo (RJMCMC) scheme where one proposes to add, delete, or move a knot during each iteration. The coefficients for the splines are the Least Square estimates.

Free Knot Splines

In my opinion this makes it only 'partially Bayesian' because for a 'fully Bayesian' approach priors would need to be placed on these coefficients (and new coefficients proposed during each iteration), but then the Least Squares estimates do not work for the RJMCMC scheme, and things become much more difficult.

$\endgroup$
  • $\begingroup$ (+1) I do not understand your situation but it seems to be a situation of a pseudo-Bayesian approach $\endgroup$ – Stéphane Laurent Jul 8 '12 at 12:50
1
$\begingroup$

I would add a characterization that has not been mentioned so far. A fully Bayesian approach "fully" propagates the uncertainty in all the unknown quantities through the Bayes theorem. On the other hand, Pseudo-Bayes approaches such as empirical Bayes do not propagate all the uncertainties. For example, when estimating posterior predictive quantities, a fully Bayesian approach would make use of the posterior density of the unknown model parameters to obtain the predictive distribution for the target parameter. An EB approach would not account for the uncertainty in all the unknowns - for example, some of the hyper-parameters may be set to particular values, thus underestimating the overall uncertainty.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.