2
$\begingroup$

I have two graphical models that have significant overlap in the random variables they are modeling. Is it possible to use samples from one graphical model to avoid sampling as much from the other graphical model?

As an example can I use samples from an empirical bayes approach in estimating the distribution in a full bayesian treatment? That is can similar models pool their samples?

If this is possible? Does this procedure allow me to use less samples when trying to simulate the full bayesian treatment? Has this been proved in a paper?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

You are mixing three unrelated concepts: empirical Bayes, bootstrapping and MCMC. The first two items are statistical procedures, while the third one is a simulation procedure. Thus, you cannot take a statistical procedure to reply to a simulation question and vice-versa. Here are some elements of answer:

  1. If you have an MCMC outcome from an empirical Bayes approach, this means you have simulated a sample of parameters for a given (posterior) distribution, $\pi_1$. If you want to get the answer for the "full Bayesian treatment", it means you are changing the target (posterior) distribution, to $\pi_2$. (The fact that one is empirical and the other fully Bayes does not matter at this level.) Therefore, if you weight your original sample $\theta_1,\ldots,\theta_T$ with importance weights $$ \omega_i=\pi_2(\theta_i)/\pi_1(\theta_i), $$ you get a weighted sample from $\pi_2$ by a standard importance sampling argument. Further, the ratio $\pi_2(\theta_i)/\pi_1(\theta_i)$ is actually the ratio of the priors, because the likelihoods are the same. (This technique only operates if you can numerically compute both priors or find some unbiased estimator of the ratio.)

  2. The bootstrap is a (neat) robust non-parametric technique that allows for a consistent approximation to the distribution of (statistical) estimators. It is therefore entrenched in a frequentist perspective, i.e. in the sample space, and does not relate to a posterior distribution that operates on the parameter space. Unless you are thinking of an advanced version of the bootstrap, it has nothing to do with MCMC (post) processing. Two entries you should consult are (a) Donald Rubin's Bayesian Bootstrap (Annals of Statistics, 1981) and (b) Brad Efron's tech report The bootstrap and Markov chain Monte Carlo (that you could have found by googling "Bayesian boostrap" and "MCMC bootstrap").

  3. Your original problem ("I have two graphical models that have significant overlap...") sounds to be more challenging because the models do not seem to share exactly the same variables/parameters. In this case, importance sampling cannot operate. (And I do not see how bootstrap would help there.)

Improve this answer
$\endgroup$
2
  • $\begingroup$ I misused the word bootstrap. I mean to ask is there a way I can reuse samples from another graphical model if they share random variables? $\endgroup$
    – zaxtax
    Nov 24, 2011 at 9:44
  • $\begingroup$ You have to be more precise in explaining how "they share random variables": if they have exactly the same random variables $X_1,\ldots,X_p$, then, as I tried to explain above, you can recyle your sample from the first graphical model via an importance sampling step. $\endgroup$
    – Xi'an
    Nov 25, 2011 at 8:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.