I want to sample from a distribution with density $$ f(\mathbf x) = \int f(c) \prod_{i=1}^n f(x_i|c) dc $$ where $\mathbf x=(x_1,x_2,...,x_n)$. In my particular setup, is easy to sample according to the densities $f(c)$ and $f(x_i|c)$, but it is not obvious how to sample from the marginal given above.

Would the following work?

  1. Sample $c$ according to $f(c)$.
  2. Sample $x_i$ according to $f(x_i|c)$, for $i=1,2,...,n$.

Heuristic argument for why this should work:

  • Obviously $f(\mathbf x,c) = f(\mathbf x|c)f(c)$, so the $(\mathbf x,c)$ pairs generated by my procedure above are distributed according to $f(\mathbf x,c)$. Marginalization amounts to ignoring the $c$'s in any pair $(\mathbf x,c)$. So the $\mathbf x$'s produced in step 2 of my procedure, taken by themselves, are distributed according to $f(\mathbf x)$, as desired.

Is this sensible? Is there a short way to make it rigorous if so? Thanks!

  • $\begingroup$ what you say makes sense. But if you want to sample $c$ from the posterior distribution $[c|x]$, then your strategy won't work. $\endgroup$ – suncoolsu Jul 19 '11 at 3:27
  • $\begingroup$ Oops, I think I used the wrong terminology. I meant "marginal". I'll update the question. $\endgroup$ – Mhc Jul 19 '11 at 3:35
  • $\begingroup$ Oh! I almost wrote the wrong reply then! $\endgroup$ – suncoolsu Jul 19 '11 at 3:44

You are right!

The formulation you describe has a general name to it. The theorem is called de-Finetti's theorem for exchangeable sequences, and is the fundamental theorem behind the Bayesian philosophy and ideas. Specifically, $x_i$s are exchangeable, that is, any permutation of $x_i$s will have the same distribution as $f(x_1,\ldots,x_n)$.

In your strategy, I would be a bit more comfortable if you had drawn $c$ for every $i$. That is, generate $c$ from $f(c)$ and then generate $x_i$, and repeat this for $i=1\ldots n$. But I think if you generate enough $\mathbf {x}$s, this should not be a problem.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.