2
$\begingroup$

Suppose I have a target distribution $\pi(\theta|x) \propto P(x|\theta)P_{\theta}(\theta)$ (e.g. the unnormalized posterior). I would like to use rejection sampling to obtain many samples $\{\theta_i\}$ from the posterior. Rejection sampling requires the use of a proposal distribution, $g(\theta)$, such that $c \cdot g(\theta) \geq \pi(\theta|x), \ \forall \ \theta$ (so $c\cdot g(\theta)$ envelopes the unnormalized posterior).

My question is: Can I get samples from $\pi(\theta|x)$ by setting $g(\theta) = P_{\theta}(\theta)$? In other words, can my proposal distribution be my prior (times a constant), which I then compare to the likelihood $P(x|\theta)$? e.g. my proposed algorithm is as follows:

  1. Sample $\theta_i$ from $P_\theta(\theta)$
  2. Sample $U_i$ from $Unif(0,1)$
  3. If $U_i \cdot (C \cdot P_\theta(\theta_i)) \leq P(x|\theta_i)$, accept $\theta_i$. Otherwise reject.
  4. Repeat many times

Will the resulting samples follow the posterior $\pi(\theta|x)$? Thank you!

$\endgroup$
3
  • $\begingroup$ I'd be more likely to try to do something with an approximation to the likelihood, but in some cases you could. Keep in mind that the prior must be heavy tailed enough that a multiple of it is larger than the distribution you want to sample everywhere, but at the same time you need $c$ not to be too large. Some priors could work okay for that (with some models at least), others won't. $\endgroup$
    – Glen_b
    Aug 11, 2021 at 3:21
  • $\begingroup$ @Glen_b Thanks, that makes sense. Still, would I be obtaining samples from the posterior or the likelihood with my algorithm above? That's a bit unclear to me at the moment... $\endgroup$ Aug 11, 2021 at 3:24
  • $\begingroup$ Ah, sorry, didn't even look at the RHS, given the title. You should be comparing with the target, since that's what you want to sample from. $\endgroup$
    – Glen_b
    Aug 11, 2021 at 3:36

1 Answer 1

1
$\begingroup$

I will use the alternative notation $\pi_0$ for the prior, since this is less confusing.

If you want to obtain samples from the posterior then you need to set this as your target distribution, not the sampling density. This requires that there exists a value $C \in \mathbb{R}$ such that $\pi(\theta|x) \leqslant C \pi_0(\theta)$ for all $\theta \in \Theta$. Assuming this holds, your acceptance indicator should be:

$$\mathscr{A}_i(\theta) \equiv \mathbb{I} \Bigg( U_i \leqslant \frac{\pi(\theta|x)}{C \cdot \pi_0(\theta)} \Bigg).$$

Using this acceptance event we get the conditional distribution:

$$\begin{align} p(\theta_i | \mathscr{A}_i) &= \frac{p(\theta_i, \mathscr{A}_i(\theta_i)=1)}{\mathbb{P}(\mathscr{A}_i(\theta_i)=1)} \\[6pt] &= \frac{\mathbb{P}(U_i \cdot C \cdot \pi_0(\theta_i) \leqslant \pi(\theta_i|x)) \cdot \pi_0(\theta_i)}{\int \mathbb{P}(U_i \cdot C \cdot \pi_0(\theta) \leqslant \pi(\theta|x)) \cdot \pi_0(\theta) \ d\theta } \\[6pt] &= \frac{\pi(\theta_i|x)/ C}{\int \pi(\theta|x)/C \ d\theta } \\[6pt] &= \frac{\pi(\theta_i|x)}{\int \pi(\theta|x) \ d\theta } \\[6pt] &= \pi(\theta_i|x), \\[6pt] \end{align}$$

which establishes the desired target distribution. Now, so long as the required bound shown above holds, it is possible to perform rejection sampling using this method to get posterior samples. However, even in this case, crude rejection sampling is quite inefficient compared to standard MCMC methods (e.g., Metropolis-Hastings, etc.).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.