# How to include prior information about target pdf in MCMC

Is there a somewhat principled way to include prior information about a target density $f(x)$ in a sampling (MCMC) algorithm? [This is a much better formulated version of this question, which I am going to close/delete.]

Suppose you want to sample from an (unnormalized) target pdf $f(x)$, $x \in \mathbb{R}^d$, and for simplicity we assume a low-dimensional $d \le 20$. $f(x)$ can be mildly expensive to evaluate (e.g., you could evaluate a simple surrogate $10^2\sim10^3$ times in the time it takes to evaluate $f$).

You have some prior information about $f(x)$. This information can be expressed in the following form:

• An approximated pdf $g \approx f$ (for concreteness, $g$ could be a mixture of Gaussians or $t$ distributions; you can think of it as a point estimate of a pdf over $f$); or
• A distribution over $f$ (represented, e.g., as a variational finite mixture, or variational DP mixture).

In both cases, we have the advantage that we can directly take samples from $g(x)$ or $\left\langle g(x) \right\rangle$. (We could use other more flexible forms to express a prior over $f$, such as a square-root GP, but then we would lose the ability to directly sample from it, and GPs become unwieldy very quickly).

In my previous, not that well-formulated question I suggested that clearly one can use $g(x)$ (or $\left\langle g(x) \right\rangle$) as a proposal for an independent Metropolis sampler. This is going to work well if our prior information is very tight about $f$, but it can clearly become very inefficient (e.g., prior information might be good for the bulk but vague or unreliable for the tails of $f$).

I can imagine various hacks or ad-hoc ways to incorporate bits and pieces of $g$ into the sampling (as some form of surrogate function, or as a testbed to tune hyperparameters for transition operators to deploy on $f$), but I am wondering if there is some approach that is more principled (or more effective) than others.

PS: As brought up in the answer by @pehup, one of the things I have been doing in practice, and that seems to work well, is alternating between an independent Metropolis proposal and a set of a few local transition operators (in particular, I am using slice sampling with random direction and window size chosen according to global information from $g$, and I am playing around with similar ideas). However, this seems like a hack -- I can see why it is a sensible thing to do, but it doesn't feel particularly principled.

• I find hard to do literature search about this topic because "prior information" gets confounded with typical Bayesian priors over $x$, but here I want to express a prior over $f$. The only somewhat related topic I could find is Bayesian Monte Carlo, also known as Bayesian quadrature (BQ), but the point of BQ is to evaluate a given integral, as opposed to obtain samples from a pdf. – lacerbi May 26 '16 at 13:54