Variance of Monte Carlo integration with importance sampling

Question

I am following these lecture slides on Monte Carlo integration with importance sampling. I am just implementing a very simple example: $\int_{0}^{1} e^{x}dx$. For the importance sampling version, I rewrite $\int_{0}^{1} e^{x}dx = \int_{0}^{1} e^{x}/p(x)\cdot p(x)dx$ where $p(x) = 2.5x^{1.5}$. Then

$$\hat{I} = \frac{1}{N}\sum_{j=1}^{N} \frac{f(x_{j})}{p(x_{j})},$$

where $x_{j}$ are sampled from $p(x_{j})$ (I use an inverse transform method here). For the variance, I have $\sigma_{I}^{2} = \hat{\sigma}_{I}^{2}/N$ and

$$\hat{\sigma}_{I}^{2} = \frac{1}{N} \sum_{j=1}^{N} \frac{f(x_{j})^{2}}{g(x_{j})^{2}} - \hat{I}^{2}.$$

I know I should expected the variance to decrease with importance sampling, but a plot of the variance with $N$ shows that not much happens. Can anyone explain to me what I'm doing incorrectly? I'm not sure how the they are able to achieve such a drastic decrease in variance in the lecture slides.

Answer 1

This is a good illustration of the dangers of importance sampling: while $$\int_0^1 \frac{e^x}{p(x)}\, p(x)\text{d} x = \int_0^1 e^x \text{d} x = I$$ shows that $\hat{I}_N$ is an unbiased estimator of $I$, this estimator does not have a finite variance since $$\int_0^1 \left(\frac{e^x}{p(x)}\right)^2\, p(x)\text{d} x = \int_0^1 \frac{e^{2x}}{2.5 x^{1.5}} \text{d} x = \infty$$ since the integral diverges in $x=0$. For instance,

> x=runif(1e7)^{1/2.5}
> range(exp(x)/x^{1.5})
[1]     2.718282 83403.685972

shows that the weights can widely differ. I am not surprised at the figures reported in the above slides since

 > mean(exp(x)/x^{1.5})/2.5
 [1] 1.717576
 > var(exp(x)/x^{1.5})/(2.5)^2/1e7
 [1] 2.070953e-06

but the empirical variance is rarely able to spot infinite variance importance sampling. (The graph shows that both the standard Monte Carlo estimate and the importance sampling version see the empirical standard deviation is decreasing as $N^{-1/2}$.)