A surprising results is that MCMC sampling remains unbiased if instead of computing the likelihood $p(x)$ ones compute an estimate $\hat{p}(x)$ as long as $\hat{p}$ is an unbiased and positive estimator of $p(x)$. This result is used to calibrate HMM by marginalizing the likelihood using a particle filter.
Suppose however that we do not have an unbiased estimate of $p(x)$ but an unbiased estimate of $\hat{lp}(x)$ of $\log p(x)$. Such a case naturally arises when we consider the likelihood of a model given an i.i.d sample. When the sample is large, it would be of great interest to quickly approximate the log likelihood at each state by only computing it for a small random sub-sample.
However, we have a biased estimate, and we don't even know its distribution. If the sub-sample is large enough, we could reasonably invoke the central limit theorem, estimate the standard deviation $\hat{\sigma}$ and apply a convexity adjustment of $e^{\hat{\sigma}^2/2}$.
However, there may be tricks I'm missing that would allow us to recover unbiased sampling. In particular, instead of using a fixed, random sub-sample, it might be possible to follow a stochastic procedure to select the sub-sample in a way that preserves the property sought.
Are there any paper that explore this problem, or any obvious solution I'm overlooking?
