2
$\begingroup$

I want to sample from the posterior density $f(x \mid y)$ which is related to the prior density $f(x)$ and likelihood density $f(y \mid x)$ via Bayes' rule: $$ f(x \mid y) = \frac{f(y \mid x) f(x)}{c} $$ where $c$ is a normalizer. Let's say both the prior and likelihood are very simple and I can sample from them directly using standard techniques, e.g. Gaussian. Specifically, say $X \sim \mathcal{N}(\mu, \sigma_1^2)$ and $Y \mid X = x \sim \mathcal{N}(x, \sigma_2^2)$. I can then sample from $f(x)$ directly, and given this sample can obtain a sample from $f(y \mid x)$ as well. Is there some method to use these samples to obtain a sample from $f(x \mid y)$? As I understand, MCMC would be used if the posterior is "difficult" to sample from, but it seems it should be easy in this case.

$\endgroup$
1
  • $\begingroup$ I believe Variational Bayes or Laplace sampling could be used here. $\endgroup$
    – user46925
    Mar 20, 2016 at 15:07

1 Answer 1

3
$\begingroup$

If $X \sim N(\mu, \sigma^2_1)$ and $Y|X = x \sim N(x, \sigma^2_2)$, then the posterior distribution $X|Y$ is also a normal distribution.

$$X|Y=y \sim N \left( \left(\dfrac{\mu}{\sigma^2_1} + \dfrac{y}{\sigma^2_2} \right)\left(\dfrac{1}{\sigma^2_1} + \dfrac{1}{\sigma^2_2} \right)^{-1}, \left(\dfrac{1}{\sigma^2_1} + \dfrac{1}{\sigma^2_2} \right)^{-1} \right) $$

So this is also easy to sample from. Such an instance of the posterior being from the same family of distributions as the prior is called conjugacy. You can see more combinations to priors and likelihoods that lead to feasible posteriors here: https://en.wikipedia.org/wiki/Conjugate_prior

$\endgroup$
3
  • $\begingroup$ Thanks for the info, but I meant for my question to be more general. That is, given the prior and likelihood are easy to sample from, is there a method to obtain a sample from the posterior besides MCMC? $\endgroup$
    – bcf
    Mar 21, 2016 at 1:11
  • 1
    $\begingroup$ The fact that sampling from the prior and likelihood is easy does not generally mean that the posterior will be easy to sample from. There is no algorithm that leads to sampling from the posterior using samples obtained from the prior and likelihood. Now, if the posterior is a simple form (like the above example) you can use direct sampling. If the posterior is complicated (intractable), you generally resort to MCMC, after trying Monte Carlo methods (rejection sampling, importance sampling etc). $\endgroup$ Mar 21, 2016 at 1:18
  • $\begingroup$ This is somewhat connected with my unanswered question on how to simulate from a geometric mixture? $\endgroup$
    – Xi'an
    Apr 7, 2016 at 11:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.