7
$\begingroup$

Say I'm able to sample an RV $X$ from a PDF $f(x)$, can I exploit this to efficiently sample another RV $Y \sim k \cdot f(g(y))$ (where $k$ is a normalizing constant)?

I'm interested in something for a Gibbs sampler that would be better than using Metropolis, slice, inverse transform sampling, etc.

$\endgroup$
  • $\begingroup$ I don't think there should be a reasonable way to do this in general with a single draw from $f$. You either need some special conditions on $g(x)$ and $f(x)$ or be willing to settle for an approximation, considering that if you could get exact samples like this in general then you could devise samplers for getting a sample from any distribution by choosing $g$ and $f$ appropriately. I'm also not hopeful for your related problem since it would imply that we could simulate from $f(\theta | x)$ if $\theta$ is real provided $f(\theta)$ is easy to sample from and $f(x|\theta)$ is symmetric. $\endgroup$ – guy Apr 14 '13 at 5:18
  • $\begingroup$ Could you post the 'related question' as a different question? (and then put that link here) $\endgroup$ – Aaron McDaid Apr 14 '13 at 13:14
  • $\begingroup$ Is $f \circ g$ similar to $f$? If so, this is easy. I guess not though. Can you tell us anything more? $\endgroup$ – Aaron McDaid Apr 14 '13 at 13:15
  • $\begingroup$ @AaronMcDaid: Good idea, done. $\endgroup$ – roger_ Apr 14 '13 at 15:48
4
$\begingroup$

If the inverse of $g(y)$, i.e. $g^{-1}(x)$, is relatively linear for the most probable values drawn from $f(x)$, then the following method should have a reasonable acceptance rate. I'm assuming $g(y)$ is strictly monotonic - in order that it have an inverse.

This is Metropolis-Hastings. We make a proposal, and then accept or reject appropriately.

Make a draw from your sampler, $x \sim f(x)$, and feed that into the inverse to get a proposal $y_{proposal} = g^{-1}(x)$. The probability of making that proposal is proportional to $$\propto \frac{f(x)}{ \left| \frac{d}{dx} g^{-1}(x) \right| } $$

That expression uses the derivative of the inverse to account for the fact that it will overrepresent areas where g^{-1} is relatively constant.

We must calculate the proposal probabilty for both the proposed value, $y_{proposal}$, and the current value, $y_{current}$. We can also use the definitions $x_{proposal} = g(y_{proposal})$ and $x_{current} = g(y_{current})$.

$$ Acceptance~probability = \operatorname{min} \left(1, \frac{ f(g(y_{proposal})) }{ f(g(y_{current})) } \frac{ \frac{f(x_{current})}{ \left| \frac{d}{dx} g^{-1}(x_{current}) \right| } }{ \frac{f(x_{proposal})}{ \left| \frac{d}{dx} g^{-1}(x_{proposal}) \right| } } \right) $$

Don't worry about the absolute value $|\cdot|$ in that equation. $g(y)$ is either increasing everywhere or decreasing everywhere and therefore the absolute values will cancel out.

We can do a lot of cancelling here, in particular remember that $x = g(y)$.

$$ Acceptance~probability = \operatorname{min} \left(1, \frac{ f(x_{proposal}) }{ f(x_{current}) } \frac{ \frac{f(x_{current})}{ \left| \frac{d}{dx} g^{-1}(x_{current}) \right| } }{ \frac{f(x_{proposal})}{ \left| \frac{d}{dx} g^{-1}(x_{proposal}) \right| } } \right) $$

$$ Acceptance~probability = \operatorname{min} \left(1, \frac{ { \frac{d}{dx} g^{-1}(x_{proposal}) } }{ { \frac{d}{dx} g^{-1}(x_{current}) } } \right) $$

Finally, I think you can rearrange a little more, and make use of the fact that the derivative of the inverse is the reciprocal of the derivative of the original function:

$$ Acceptance~probability = \operatorname{min} \left(1, \frac{ { \frac{d}{dy} g(y_{current}) } }{ { \frac{d}{dy} g(y_{proposal}) } } \right) $$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.