Given a quadratic loss function, the Bayes estimator is given by \begin{equation}\hat{\theta} = E[\theta|y] = \frac{\int_\Theta\theta p(y|\theta) p(\theta) d\theta}{\int_\Theta p(y|\theta) p(\theta) d\theta}\end{equation} where $\theta\in\Theta$ is the parameter vector, $y$ is the data, $p(\theta)$ is the prior, $p(\theta|y)$ is the posterior, and $p(y|\theta)$ is the likelihood. A variety of numerical techniques (Rejection Sampling, MCMC methods, etc.) have been developed to draw from the posterior distribution when direct sampling from the posterior is unavailable. These techniques allow us to estimate $E[\theta|y]$ with \begin{equation}E[\theta|y]\approx\frac{1}{n}\sum_{i=1}^n\theta_i\end{equation} where $\theta_i$ is the $i^{th}$ draw from the posterior.

My question: If we are able to draw from the prior distribution, couldn't we just sample the prior to numerically approximate the integrals on the right hand side of the first equation? In other words, approximate $E[\theta|y]$ with\begin{equation}E[\theta|y]\approx \frac{\frac{1}{n}\sum_{i=1}^n\theta_i p(y|\theta_i)}{\frac{1}{n}\sum_{i=1}^np(y|\theta_i)} = \frac{\sum_{i=1}^n\theta_i p(y|\theta_i)}{\sum_{i=1}^np(y|\theta_i)} \end{equation} where $\theta_i$ is the $i^{th}$ draw from the prior. This would seem to be a more convenient and computationally efficient alternative to Rejection Sampling and MCMC techniques. I understand that in some cases we may not be able to draw from the prior, but in those cases why not just pick different priors? I haven't seen the method I'm proposing discussed in any textbooks or online, so I'm hoping someone can explain to me the folly of this approach.

  • 2
    $\begingroup$ Numerical approximation is widely used compared to MCMC sampling. It is just a matter of choice really, although there are benefits and drawbacks to both. For people who aren't familiar with mathematical theories and concepts of numerical approximations, it is just simpler to understand the workings of MCMC algorithms as opposed to Newton methods and such. $\endgroup$ – Tony Hellmuth May 31 '18 at 2:48
  • $\begingroup$ Do you know of any specific instances in which people have used this method? As I said, I have yet to come across any reference that makes use of this approach. I would also argue that the method I'm proposing is much easier to understand than MCMC techniques and it does not use any Newtonian methods. $\endgroup$ – Peter May 31 '18 at 16:14
  • $\begingroup$ Yes certainly in hydrology, geo-spatial modelling and physics. There is a tendency to see computational power play a huge role since MCMC takes so freaking long to converge in say 10 parameter models and highly order differential equations when working with large, like massive datasets. Your answer below is certainly apt though, good find! $\endgroup$ – Tony Hellmuth May 31 '18 at 23:56

Alright, I think I found the answer to my question. The reason my approximation is rarely/never used is that it is generally a very poor approximation of the integrals. To see why, consider what happens when the likelihood function spikes somewhere in the tails of the prior (as will often be the case). The $\theta$'s surrounding the spike will rarely be sampled, resulting in a poor approximation of the likelihood function. See pages 25 and 26 of the document linked below for more information: https://engineering.purdue.edu/kak/Tutorials/MonteCarloInBayesian.pdf

Edit: The first paragraph of section 3.2 in the following document also discusses this but in the context of computing Bayes factors: https://www.insper.edu.br/wp-content/uploads/2014/09/2014_wpe341.pdf


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.