I want to sample from the posterior density $f(x \mid y)$ which is related to the prior density $f(x)$ and likelihood density $f(y \mid x)$ via Bayes' rule: $$ f(x \mid y) = \frac{f(y \mid x) f(x)}{c} $$ where $c$ is a normalizer. Let's say both the prior and likelihood are very simple and I can sample from them directly using standard techniques, e.g. Gaussian. Specifically, say $X \sim \mathcal{N}(\mu, \sigma_1^2)$ and $Y \mid X = x \sim \mathcal{N}(x, \sigma_2^2)$. I can then sample from $f(x)$ directly, and given this sample can obtain a sample from $f(y \mid x)$ as well. Is there some method to use these samples to obtain a sample from $f(x \mid y)$? As I understand, MCMC would be used if the posterior is "difficult" to sample from, but it seems it should be easy in this case.

  • $\begingroup$ I believe Variational Bayes or Laplace sampling could be used here. $\endgroup$ – user46925 Mar 20 '16 at 15:07

If $X \sim N(\mu, \sigma^2_1)$ and $Y|X = x \sim N(x, \sigma^2_2)$, then the posterior distribution $X|Y$ is also a normal distribution.

$$X|Y=y \sim N \left( \left(\dfrac{\mu}{\sigma^2_1} + \dfrac{y}{\sigma^2_2} \right)\left(\dfrac{1}{\sigma^2_1} + \dfrac{1}{\sigma^2_2} \right)^{-1}, \left(\dfrac{1}{\sigma^2_1} + \dfrac{1}{\sigma^2_2} \right)^{-1} \right) $$

So this is also easy to sample from. Such an instance of the posterior being from the same family of distributions as the prior is called conjugacy. You can see more combinations to priors and likelihoods that lead to feasible posteriors here: https://en.wikipedia.org/wiki/Conjugate_prior

  • $\begingroup$ Thanks for the info, but I meant for my question to be more general. That is, given the prior and likelihood are easy to sample from, is there a method to obtain a sample from the posterior besides MCMC? $\endgroup$ – bcf Mar 21 '16 at 1:11
  • 1
    $\begingroup$ The fact that sampling from the prior and likelihood is easy does not generally mean that the posterior will be easy to sample from. There is no algorithm that leads to sampling from the posterior using samples obtained from the prior and likelihood. Now, if the posterior is a simple form (like the above example) you can use direct sampling. If the posterior is complicated (intractable), you generally resort to MCMC, after trying Monte Carlo methods (rejection sampling, importance sampling etc). $\endgroup$ – Greenparker Mar 21 '16 at 1:18
  • $\begingroup$ This is somewhat connected with my unanswered question on how to simulate from a geometric mixture? $\endgroup$ – Xi'an Apr 7 '16 at 11:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.