4
$\begingroup$

$\newcommand{\E}{\mathbb{E}}$I'm reading a book on machine learning and sampling methods and I want to know why the estimator of the normalizing constant is unbiased, but the estimator of $\E\left[f(x)\right]$ is biased. Please see the image below:

enter image description here

My question is, how we can prove that importance sampling,

  1. leads to an unbiased estimator of the normalizing constant or $\E\left[\hat{Z}\right] = Z $;
  2. and the estimator of $\E\left[f(x)\right]$ for a function $f(.)$ is a biased.

Edit:

Suppose $Z$ is the normalizing constant of the desired distribution $p(x) = \phi(x)/Z$ ($p(x)$ is only known up to the normalizing constant $Z$). We have

$$I = \E\left[{\bf{f}}({\bf{x}})\right] = \frac{1}{Z}\int {{\bf{f}}({\bf{x}})\varphi ({\bf{x}})} d{\bf{x}} = \frac{1}{Z}\int {{\bf{f}}({\bf{x}})\frac{{\varphi ({\bf{x}})}}{{q({\bf{x}})}}q({\bf{x}})} d{\bf{x}}$$

and its approximation

$$I_N = \frac{{\frac{1}{N}\sum\nolimits_{i = 1}^N {{\bf{f}}({{\bf{x}}^i})w({{\bf{x}}^i})} }}{{\frac{1}{N}\sum\nolimits_{j = 1}^N {w({{\bf{x}}^j})} }} = \sum\nolimits_{i = 1}^N {{\bf{f}}({{\bf{x}}^i})W({{\bf{x}}^i})} $$

as well as

$$\hat Z = \frac{1}{N}\sum\nolimits_{i = 1}^N {w({{\bf{x}}^i})} $$

My questions are:

1- Why the author takes $Z = \int {\varphi ({\bf{x}})} d{\bf{x}}$?

2- I'm not able to prove mathematically that $\E\left[\hat{Z}\right] =Z $ ,

3- and I want to know how one can prove that $\hat{I_N}$ is biased for finite values of N?

$\endgroup$
  • $\begingroup$ Please type your questions instead of posting images. They can't be read by the visually impaired using screen readers. $\endgroup$ – gung Nov 6 '16 at 13:15
  • $\begingroup$ Possible duplicate of Optimal importance sampling with ratio estimator $\endgroup$ – Xi'an Nov 6 '16 at 13:15
  • $\begingroup$ Check also my answer there for additional solutions. $\endgroup$ – Xi'an Nov 6 '16 at 13:19
  • 1
    $\begingroup$ The answer to your second first question is clear. $\int p(x)dx = 1$ since it's a distribution fn. So it follow directly since $ \phi(x) = Z p(x) $ I don't think it's an answer (no details) but second second and third questions are both versions of the basic fact of MC theory or sampling theory, that is if one picks iid points $x_i$ the sums converge to the integral. $\endgroup$ – aginensky Nov 6 '16 at 14:14
  • $\begingroup$ Could you please indicate the source of the reference? $\endgroup$ – Xi'an Nov 6 '16 at 14:42
2
$\begingroup$
  1. Why the author takes $\mathfrak{Z}=∫φ(x)dx$?

Since $p$ is a density, its integral is equal to $1$. If $\mathfrak{Z}$ is the normalising constant of $\varphi$, it has to satisfy $$\int p(x)\text{d}x=\int \frac{\varphi(x)}{\mathfrak{Z}}\text{d}x=1$$

2- I'm not able to prove mathematically that why $\mathbb{E}[\hat{\mathfrak{Z}}]=\mathfrak{Z}$

Recall that $w(x)=\varphi(x)/q(x)$. Then $$\mathbb{E}[w(X)]=\int \frac{\varphi(x)}{q(x)}q(x)\text{d}x= \int \varphi(x)\text{d}x=\mathfrak{Z}$$

3- and I want to know how one can prove that $\hat{I_N}$ is biased for finite values of N?

The ratio of two unbiased estimators is biased since $$\mathbb{E}[1/h(X)]\ge1/\mathbb{E}[h(X)]$$by Jensen's inequality.

Note: There exist unbiased estimators of the inverse $\mathfrak{Z}^{-1}$, including the notorious harmonic mean estimator.

$\endgroup$
  • 2
    $\begingroup$ Pretty simple solutions! Merci beaucoup. $\endgroup$ – sci9 Nov 6 '16 at 14:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.