# Importance Sampling: using Target Distribution as Proposal Distribution to approximate normalizing constant

Importance Sampling is a method use to approximate expectations of a test function $$\phi$$ with respect to $$p$$ by instead sampling from a proposal distribution $$q$$ $$\mathbb{E}_{p}[\phi(x)] = \int \phi(x) p(x) dx = \frac{Z_q}{Z_p}\int\phi(x) \frac{\tilde{p}(x)}{\tilde{q}(x)} q(x)dx = \frac{\displaystyle \int \phi(x) w(x) q(x)dx}{\displaystyle \int w(x) q(x) dx} \approx \sum_{i=1}^n \phi(x^{(i)})W(x^{(i)}) \qquad x^{(i)}\sim q$$

The normalizing constant is approximated with the following unbiased estimator $$Z_p = \int \tilde{p}(x) dx = \frac{1}{n}\sum_{i=1}^n w(x^{(i)})$$

Is it possible to use $$q(x) = p(x)$$? I would like to do so in order to approximate the normalizing constant.

This doesn't seem to work because then $$w(x^{(i)}) = 1$$ and we would have $$Z_p \approx 1$$

# Update

Sorry, I forgot to mention this. I defined $$W(x) = \frac{w(x^{(i)})}{\sum_{i=1}^n w(x^{(i)})}$$ so that $$\mathbb{E}_p[\phi(x)] \approx \frac{\frac{1}{n}\sum_{i=1}^n \phi(x^{(i)}) w(x^{(i)})}{\frac{1}{n}\sum_{i=1}^n w(x^{(i)})} = \displaystyle \sum_{i=1}^n \phi(x^{(i)})\frac{w(x^{(i)})}{\sum_{j=1}^n w(x^{(j)})} = \sum_{i=1}^n \phi(x^{(i)}) W(x^{(i)})$$

• @Xi'an I have updated the question, $W$ are the normalized importance weights! May 4, 2021 at 18:03
• @Xi'an Basically $p(x) = Z_p^{-1} \tilde{p}(x)$ and $q(x) = Z_q^{-1} \tilde{q}(x)$. I can compute $p$ up to a normalizing constant (i.e. I can compute $\tilde{p}$). May 4, 2021 at 18:05
• I can also sample from $p(x)$ May 4, 2021 at 18:15

If $$p(x)=\tilde p(x)/Z_p$$, computing $$Z_p$$ using uniquely a sample from $$p$$ is not feasible by importance sampling: $$\int \tilde p(x) \,\text dx=\int \frac{\tilde p(x)}{p(x)} p(x)\,\text dx=Z_p\int p(x)\,\text dx= Z_p$$ does not help since the importance weight $${\tilde p(x)}\big/{p(x)}$$ is equal to $$Z_p$$, unknown!
When looking at either $$Z_p$$ or at the inverse of $$Z_p$$, the impossibility of finding unbiased estimators is clear: if $$\mathbb E_{p}[\ell(X)]={Z_p}\quad \text{and}\quad \mathbb E_{p}[h(X)]=\frac{1}{Z_p}$$ then \begin{align}\int \ell(x) p(x)\,\text dx&=\int\ell(x) \frac{\tilde p(x)}{Z_p}\,\text dx= {Z_p}\\ \int h(x) p(x)\,\text dx&=\int h(x) \frac{\tilde p(x)}{Z_p}\,\text dx= \frac{1}{Z_p}\end{align} implies that $$\int \ell(x) \tilde p(x)\,\text dx=Z_p^2\quad \text{and}\quad\int h(x) \tilde p(x)\,\text dx=1$$ both of which cannot hold since $$\tilde p(\cdot)$$ is known up to a constant.
This difficulty is discussed for MCMC samples in a 2008/9 BA paper I wrote with Jean-Michel Marin. And solutions are compared in this 2009/10 survey of ours. All based on additional samples$$^1$$ from different distributions (except for the infamous harmonic mean estimator!). See also the solutions based on reverse logistic regression (Geyer, 1991/4), noise contrastive estimation (Guttmann and Hyvärinen, 2010/2), sequential Monte Carlo, path sampling, &tc. (Also discussed in this X validated question.)
$$^1$$The closest to an exception may be Chib's method (or the candidate formula, cf. Besag) since the representation of the marginal is based on a single sample from the posterior, but this is an augmented posterior based on latent variables.
• Ahhh I forgot $Z_p$!! May 4, 2021 at 18:45