In a lecture recently our lecturer described a method for approximating the expectation of a function over a posterior distribution using likelihood importance sampling. That is:

$$ \mathbb{E}_{p(x|D)}[f(x)] \text{ can be approximated by } \frac{1}{\sum_{n=1}^{N}p(D|x^{(n)})}\sum_{n=1}^{N}p(D|x^{(n)})f(x^{(n)})$$

where $x^{(n)}$ are i.i.d. RV sampled from the prior $p(x)$, and $p(D|x^{(n)})$ is the likelihood of the observed data $D$.

He then claims that this estimator is biased but consistent. How can I see this? To see it is biased you usually find the expectation of the estimator and I cannot see how to do that here

$$ \mathbb{E}[\frac{1}{\sum_{n=1}^{N}p(D|x^{(n)})}\sum_{n=1}^{N}p(D|x^{(n)})f(x^{(n)})]$$ as the denominator and numerator are non-independent functions of $x$. Am I missing something and the denominator can be considered a constant and taken out of the expectation?

Any help or intuition much appreciated.

  • $\begingroup$ A sample from the prior need be weighted by the likelihood to become a sample from the posterior. That it is biased is due to the division by the average$$\frac{1}{N}\sum_{n=1}^{N}p(D|x^{(n)})$$whose expectation is the normalising constant (or marginal likelihood) and hence whose inverse does not have as expectation the inverse of the normalising constant. $\endgroup$ – Xi'an Oct 27 at 21:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.