3
$\begingroup$

I have a Bayesian model with a large number of parameters (around 50), and as usual my goal is to infer the posterior distribution for the parameters, with MCMC.

However, I am only interested in the full posterior distribution for 5 of the parameters - the others are indispensable to the model but uninteresting. Due to the impossible computational cost of th MCMC chain on all 50 parameters, I am looking at this method to give some idea of the posteriors I am interested in, while still being possible to run in the amount of time:

  1. Obtain the MAP estimate of all 50 parameters.
  2. Obtain the MCMC samples of the posterior distribution for the 5 interesting parameters, conditional on the MAP estimate for the other parameters.

Obviously this conditional posterior is not as good as the correct joint posterior. But as an approximation, do you think the strategy is plausible? Its okay to assume that the parameters using MAP estimates dont have alot of variance.

$\endgroup$
1
  • $\begingroup$ It is hard to tell as it depends on the problem. A minima, repeated experiments with different values of the plug-in estimates would be needed to assert the impact. $\endgroup$ – Xi'an May 28 '20 at 15:17
3
$\begingroup$

Since you are interested in simulating $$\pi(\theta_1|\mathbf x) = \int \pi(\theta_1,\theta_2|\mathbf x)\,\text{d}\theta_2$$ you are essentially seeking a manageable approximation to this integral that does not involve simulating the joint $\pi(\theta_1,\theta_2|\mathbf x)$. The MAP proposal is stating that $$\int \pi(\theta_1,\theta_2|\mathbf x)\,\text{d}\theta_2\approx \pi(\theta_1|\hat\theta_2^\text{MAP},\mathbf x)$$ for all $\theta_1$'s which is quite crude. Note that there are two possible choices for the MAP estimate, one being the joint MAP and the other the marginal MAP, presumably impossible to derive.

A less crude version would be to use a Laplace approximation of this integral, replacing $\pi\theta_2|\mathbf x)$ with a Normal centered at the MAP estimate and a variance covariance matrix associated with the Fisher information (or its observed version), $\mathcal N(\hat\theta_2^\text{MAP},\hat\Sigma_2)$. The integral could then be approximated by $$\int \pi(\theta_1,\theta_2|\mathbf x)\,\text{d}\theta_2\approx \frac{1}{N}\sum_{i=1}^N \pi(\theta_1|\theta_2^{(i)},\mathbf x)\qquad\theta_2^{(i)}\sim\mathcal N(\hat\theta_2^\text{MAP},\hat\Sigma_2)$$ A presumably better (and unbiased) approximation is to resort to importance weights $$\int \pi(\theta_1,\theta_2|\mathbf x)\,\text{d}\theta_2\approx \frac{1}{N}\sum_{i=1}^N \frac{\pi(\theta_1|\theta_2^{(i)},\mathbf x)}{\varphi(\theta_2^{(i)}|\hat\theta_2^\text{MAP},\hat\Sigma_2)}\qquad\theta_2^{(i)}\sim\mathcal N(\hat\theta_2^\text{MAP},\hat\Sigma_2)$$ where $\varphi(\theta_2^{(i)}|\hat\theta_2^\text{MAP},\hat\Sigma_2)$ denotes the density of the approximating Normal distribution. A more involved version of this idea is to use integrated nested Laplace approximation (INLA), available in some pseudo-Gaussian settings. (Note that any importance function substitute could be used in the above.)

Note also that Chen, Shao & Ibrahim (1999) have an entire chapter dedicated to the approximation of marginal posterior densities, which may be of help.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.