Is there a somewhat principled way to include prior information about a target density $f(x)$ in a sampling (MCMC) algorithm? [This is a much better formulated version of this question, which I am going to close/delete.]
Suppose you want to sample from an (unnormalized) target pdf $f(x)$, $x \in \mathbb{R}^d$, and for simplicity we assume a low-dimensional $d \le 20$. $f(x)$ can be mildly expensive to evaluate (e.g., you could evaluate a simple surrogate $10^2\sim10^3$ times in the time it takes to evaluate $f$).
You have some prior information about $f(x)$. This information can be expressed in the following form:
- An approximated pdf $g \approx f$ (for concreteness, $g$ could be a mixture of Gaussians or $t$ distributions; you can think of it as a point estimate of a pdf over $f$); or
- A distribution over $f$ (represented, e.g., as a variational finite mixture, or variational DP mixture).
In both cases, we have the advantage that we can directly take samples from $g(x)$ or $\left\langle g(x) \right\rangle$. (We could use other more flexible forms to express a prior over $f$, such as a square-root GP, but then we would lose the ability to directly sample from it, and GPs become unwieldy very quickly).
In my previous, not that well-formulated question I suggested that clearly one can use $g(x)$ (or $\left\langle g(x) \right\rangle$) as a proposal for an independent Metropolis sampler. This is going to work well if our prior information is very tight about $f$, but it can clearly become very inefficient (e.g., prior information might be good for the bulk but vague or unreliable for the tails of $f$).
I can imagine various hacks or ad-hoc ways to incorporate bits and pieces of $g$ into the sampling (as some form of surrogate function, or as a testbed to tune hyperparameters for transition operators to deploy on $f$), but I am wondering if there is some approach that is more principled (or more effective) than others.
PS: As brought up in the answer by @pehup, one of the things I have been doing in practice, and that seems to work well, is alternating between an independent Metropolis proposal and a set of a few local transition operators (in particular, I am using slice sampling with random direction and window size chosen according to global information from $g$, and I am playing around with similar ideas). However, this seems like a hack -- I can see why it is a sensible thing to do, but it doesn't feel particularly principled.
