Calculating Integral Using MCMC

Question

Consider the integral $\int_{\Theta}f(\theta|\mathbf{x}) \Pi(\theta)d\theta$,where $\theta$ is a univariate parameter and $\Theta$ is the support of $\Pi(\theta)$. I need to evaluate the value of this integral numerically. I understand that I need to draw random sample from the Prior and aggregate the value of the likelihood function but my prior is not proper so generating from it is not possible. In such cases how can I evaluate this integral?

Answer 1

First, if the prior is improper, the marginal likelihood $$\int_{\Theta}f(\theta|\mathbf{x}) \pi(\theta)\,\text{d}\theta\stackrel{\text{def}}{=}m(x)$$ is not a density. Hence, its use is restricted to normalising the posterior $$f(\theta|\mathbf{x}) \pi(\theta)\Big/\int_{\Theta}f(\theta|\mathbf{x}) \pi(\theta)\,\text{d}\theta$$ and cannot be used for testing (e.g., in a Bayes factor).

Second, there is no reason to simulate from the prior to approximate this integral (and no reason to automatically resort to MCMC). Indeed, writing $$f(\theta|\mathbf{x}) \pi(\theta) \stackrel{\text{def}}{=} h(\theta;\mathbf{x})$$ this product $h(\theta;\mathbf{x})$ is all that matters for computing the integral. Any density [in $\theta$] $g(\theta;\mathbf{x})$ with support $\Theta$ and possibly depending on $\mathbf{x}$ can thus be used to represent the integral as $$\int_{\Theta}f(\theta|\mathbf{x}) \pi(\theta)\,\text{d}\theta=\int_{\Theta}\frac{h(\theta;\mathbf{x})}{g(\theta;\mathbf{x})}\,g(\theta;\mathbf{x})\,\text{d}\theta$$ This is the principle behind the importance sampling approach.

Note that one popular (?) approach consists in using a sample from the posterior, $\theta_1,\ldots,\theta_T$, obtained for instance by an MCMC algorithm, and to exploit the identity $$\int_{\Theta}\frac{1}{f(\mathbf{x}|\theta)}\,\pi(\theta|\mathbf{x})\,\text{d}\theta=\int_{\Theta}\frac{h(\theta;\mathbf{x})}{f(\mathbf{x}|\theta)}\,\frac{1}{m(\mathbf{x})}\,\text{d}\theta=\frac{1}{m(\mathbf{x})}$$ to derive the harmonic mean estimate $$\hat{m}(\mathbf{x})^{-1}=\frac{1}{T}\sum_{t=1}^T\, f^{-1}(\mathbf{x}|\theta_t)$$ This estimator should not be used. It indeed most often enjoys an infinite variance.