(After some pondering, what I really wanted to ask is how to incorporate prior information about $f$ into a sampling method - see this question.)
Suppose you want to draw samples from an (unnormalized) pdf $f(x)$, $x \in \mathbb{R}^d$, $1 \le d \le 20$, which might be expensive to compute, and you have a (normalized) pdf, $g(x)$, which is an approximation of $f(x)$. We also assume that you can easily draw i.i.d. samples from $g(x)$, and $g(x)$ has a simple analytical form (e.g., we can derive gradients). As a practical example, let us assume that $g(x)$ is a mixture of Gaussians or $t$ distributions.
Note that $f(x)$ is black-box, so we do not know much about it besides assuming that $g(x)$ is a "decent" approximation thereof (which I deliberately leave vague).
There are several ways to proceed here:
- A standard approach would consist of using $g(x)$ as a proposal distribution of an independent Metropolis MCMC sampler (perhaps after "fattening" the tails of $g(x)$). Clearly, the acceptance/reject ratio would be based on actual values of $f(x)$.
- Some mix of importance sampling and independent Metropolis, such as the approach described in this paper (Edit: I originally mistakenly wrote "importance sampling/resampling", but that would not produce unbiased samples from $f(x)$).
- We could use $g(x)$ as a surrogate function and perform Hamiltonian Monte Carlo (or other efficient transition operators) on the surrogate, with acceptance/rejection based on the true $f(x)$.
I am trying to figure out whether there would be reasons or situations in which to prefer (3) to (1), or other methods to (1). Any idea?
A note after the discussion in the comments. The acceptance ratio of an independent Metropolis proposal from $g$ is: $$a\left(x_\text{old} \rightarrow x_\text{new}\right) = \min \left(1, \frac{f(x_\text{new})}{f(x_\text{old})} \cdot \frac{g(x_\text{old})}{g(x_\text{new})} \right). $$ In the limit $g \rightarrow f$ we have that $a \rightarrow 1$. This suggests that if $g$ is a very good approximation to $f$, one can't do much better than (1). However, if $g$ is an okay approximation to $f$, there might be better things to do.