3
$\begingroup$

Does the increase in importance sampling size guarantee the decrease in importance sampling variance?

Some context here: I'm trying to use importance sampling instead of equal probability sampling to reduce the variance of my estimator. Based on my data, if I have a small sample size, the variance of importance sampling data is smaller than the variance of equal probability sampling. But if I increase to a big sample size, equal probability sampling variance is guaranteed to be smaller, and the variance of importance sampling doesn't feel like so.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

The variance of the estimator in importance sampling still reduces in inverse-proportion to the number of values used in the simulation (assuming the variance is finite). In importance sampling, we estimate a moment quantity $\mu \equiv \mathbb{E}(h(X))$ with $X \sim f$ by generating a set of random variables $Y_1,Y_2,...,Y_m \sim \text{IID }g$ and using the estimator:

$$\hat{\mu}_m \equiv \frac{1}{m} \sum_{i=1}^m \frac{h(Y_i) f(Y_i)}{g(Y_i)}.$$

With a bit of algebra (see below) the variance of the estimator can be found to be:

$$\mathbb{V}(\hat{\mu}_m) = \frac{1}{m} \bigg[ \mathbb{E} \bigg( h(X)^2 \cdot \frac{f(X)}{g(X)} \bigg) - \mathbb{E}(h(X))^2 \bigg].$$

As you can see, this quantity is still proportionate to $1/m$ (assuming this variance is finite) so the large-simulation mechanics of this method are still essentially the same as in the unweighted case.

Deriving the variance of the estimator: Write the estimator as:

$$\hat{\mu}_m = \frac{1}{m} \sum_{i=1}^m \varepsilon(Y_i) \quad \quad \quad \quad \quad \varepsilon (y) \equiv \frac{h(y) f(y)}{g(y)}.$$

Since each of these terms is an unbiased estimator of the moment of interest, they have variance:

$$\begin{align} \mathbb{V}(\varepsilon(Y_i)) = \mathbb{E}(\varepsilon(Y_i)^2) - \mu^2 &= \mathbb{E} \bigg( \bigg( \frac{h(Y_i) f(Y_i)}{g(Y_i)} \bigg)^2 \bigg) - \mu^2 \\[6pt] &= \int \limits_\mathbb{R} \bigg( \frac{h(y) f(y)}{g(y)} \bigg)^2 g(y) \ dy - \mu^2 \\[6pt] &= \int \limits_\mathbb{R} \frac{h(x)^2 f(x)}{g(x)} f(x) \ dx - \mu^2 \\[6pt] &= \mathbb{E} \bigg( h(X)^2 \cdot \frac{f(X)}{g(X)} \bigg) - \mathbb{E}(h(X))^2 \\[6pt] \end{align}$$

We then have:

$$\begin{align} \mathbb{V}(\hat{\mu}_m) &= \mathbb{V} \bigg( \frac{1}{m} \sum_{i=1}^m \varepsilon(Y_i) \bigg) \\[6pt] &= \frac{1}{m^2} \sum_{i=1}^m \mathbb{V} ( \varepsilon(Y_i) ) \\[6pt] &= \frac{1}{m^2} \sum_{i=1}^m \bigg[ \mathbb{E} \bigg( h(X)^2 \cdot \frac{f(X)}{g(X)} \bigg) - \mathbb{E}(h(X))^2 \bigg] \\[6pt] &= \frac{1}{m} \bigg[ \mathbb{E} \bigg( h(X)^2 \cdot \frac{f(X)}{g(X)} \bigg) - \mathbb{E}(h(X))^2 \bigg]. \\[6pt] \end{align}$$

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I think it is important to note that in the derivation we sometimes take the expectation over f and sometimes over g. Would be nice to add that as a subscript. $\endgroup$
    – Jannis
    Dec 16, 2022 at 6:57
  • 1
    $\begingroup$ just to note for anyone who is confused by the notation, and would like to have subscripts on the $\mathbb E$ operators as @Jannis is asking for, you can supply those subscripts mentally: when the expression contains variable Y, use subscript g, and when X use f. $\endgroup$
    – postylem
    Feb 12 at 15:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.