I am trying to recreate the Bayesian Hierarchical Clustering algorithm using Python. The example in section two requires evaluating the following double integral (univariate case):

\begin{align} p(D_k|H_k) &= \int_\theta p(D_k | \theta) p(\theta | \theta_0) d\theta \\ &= \int_\mu \int_\phi \prod_{i = 1}^n \mathcal{N}(x_i | \mu, \phi) \mathcal{N}(\mu | \mu_0, \tau^2) \mathcal{G}(\phi | \alpha, \beta) \thinspace d\phi \thinspace d\mu \\ &= \int_\mu \int_\phi \prod_{i = 1}^n \frac{\phi^{\frac{1}{2}}}{ \sqrt{2 \pi} } \exp \left({\frac{\phi(x_i - \mu)^2}{-2}} \right) \frac{\beta^\alpha}{\Gamma(\alpha)}{\phi^{\alpha - 1} } e^{ - \beta \phi} \frac{\tau^{\frac{1}{2}}}{ \sqrt{2 \pi} } \exp \left({\frac{\tau(\mu - \mu_0)^2}{-2}} \right) \thinspace d\phi \thinspace d\mu \\ &= \int_\mu \left[ \int_\phi \prod_{i = 1}^n \frac{\phi^{\frac{1}{2}}}{ \sqrt{2 \pi} } \exp \left({\frac{\phi(x_i - \mu)^2}{-2}} \right) \frac{\beta^\alpha}{\Gamma(\alpha)}{\phi^{\alpha - 1} } e^{ - \beta \phi} \thinspace d\phi \thinspace \right] \frac{\tau^{\frac{1}{2}}}{ \sqrt{2 \pi} } \exp \left({\frac{\tau(\mu - \mu_0)^2}{-2}} \right) \thinspace \thinspace d\mu \\ \end{align}

Note that $\beta$, $\alpha$, $\mu_0$, and $\tau$ are priors that I have set (i.e. they are known).

I think it is best to use importance sampling here starting with the integral with respect to $\phi$. I am teaching myself importance sampling, but I do not understand a few things for this case:

1) How do I make sure the capture the $x_i$ product term in the inner integral? Do I need to evaluate the density at sampled values of \mu and \phi? If so, what is the best method to taking these samples and then performing the following importance sampling?

2) More broadly, is importance sampling in fact the best method for computing this integral? Is there a way to check that my results are valid for a complicated expression like this one?

Thank you for your time.


I have used the notion of Normal-Gamma conjugacy to reduce the integral above to the following:

$$ \int_\mu \int_\phi \mathcal{G}(\phi | \frac{n + \alpha}{2}, \frac{\beta \Sigma_i(x_i - \mu)^2}{2}) \thinspace d\phi \mathcal{N}(\mu | \mu_0, \tau^2) \thinspace d\mu \\ $$

But I am stil unsure about the importance sampler due to the unknown $\mu$.

  • $\begingroup$ This is a marginal likelihood. There are entire books dedicated to this problem, like Chen, Shao & Ibrahim (2000).. $\endgroup$ – Xi'an Apr 18 at 19:37
  • $\begingroup$ Note also that the exact integral in $\mu$ is available due to the conjugacy of the prior, hence no need for a double integration by importance functions. $\endgroup$ – Xi'an Apr 18 at 19:48
  • $\begingroup$ The final integral is equal to one. $\endgroup$ – Xi'an May 9 at 8:42
  • $\begingroup$ @Xi'an as this integral describes a probability distribution it seems intuitive that the final integral should equal one. Are you suggesting using a “kernel trick” to evaluate? Is there something I am missing in your most recent comment? Thanks again. $\endgroup$ – samvoit4 May 16 at 10:03
  • $\begingroup$ This is a most basic remark, I do not see the point in not integrating this density. $\endgroup$ – Xi'an May 16 at 11:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.