I have just had a course in Bayesian Inference, and I am left puzzled about what method should I actually use in practice. Assume I have a multivariate model with multiple parameters $\theta$, where all priors $p[\theta]$ are normally distributed and the likelihood $P[x|\theta]$ is not a simple function. I want to compute the model evidence $P[x]$ (to do model selection), the maximum a posteriori estimate $\theta_{MAP}$, as well as make some reasonable approximation of the posterior $P[\theta | x]$. Methods I have learned:

  1. Computing $P[x]$ numerically, e.g. using MCMC, Chib approximation, etc. Can be made to work for any problem given sufficient time. Suffers from curse of dimensionality. Does not require VB
  2. Conjugate Priors. Works for most common prior and likelihood pairs, otherwise does not work at all. Does not require VB
  3. Laplace's approximation. For simple prior and likelihood, mean and covariance are analytically computable, otherwise can use MCMC except it converges faster because we only care about mean and variance, not the integral. Does not require VB
  4. Mean-Field Approximation. Uses VB :). Approximates model evidence with negative free energy. Computes marginal distributions of parameters analytically. Then analytically computes other scary integrals to find relationships between optimal parameter values, which are then finally solved iteratively. We were not clarified what to do if any of the marginalization or optimization integrals can not be done analytically.

Questions: Is VB useful in practice? Is the additional analytic effort justified in optimizing total time to do inference (human time + machine time)? Are the mentioned integrals computable for non-basic distributions? If no, does one proceed to compute the integrals numerically? How? Is numerical VB still better than the other mentioned methods?

  • $\begingroup$ VB is, in some cases, the only feasible way to attempt inference in practice. This is because doing inference in general is a computationally hard problem, so exact inference cannot be scaled as the number of variables grows without strong restrictions on your independence structure. In terms of computing integrals, there are a wild array of MCMC methods used to approximately solve analytically intractable integrals (some of which even work well on some high dimensional spaces), and these are a staple of performing Bayesian inference. $\endgroup$ – stats_model Jun 4 at 21:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.