# Expectation of an unbiased estimator (under variational inference setting)

I am reading this ICML2016 paper, and am puzzled with the first inequality (converted to equality) on section 2.2.

Assume the model is $P(x,h)$ where $h$ are the hidden variables. Also assume $\hat{I}$ is an unbiased estimator of the likelihood term, $P(x)$. From this we can conclude that $$E_{P(x)}[\hat{I}]-P(x)=0\Rightarrow E_{P(x)}[\hat{I}] = P(x)$$ now assume we want to establish a lower bound on $P(x)$ (similar to EM approach) and plug in the estimator $\hat{I}$, instead of $P_{\theta}(x)$ in the lower bound formulation.

For this imagine the posterior distribution over the latent variables $h$, to be estimated using $Q(h|x)$ (i.e., $Q$ is a variational posterior). So if we want to write

\begin{align} &\log P(x) =\log \sum P(x,h)\\&\Rightarrow \log P(x) = \log \sum P(x,h)\frac{Q(h|x)}{Q(h|x)}\\&\Rightarrow \log E_{Q(h|x)}[P(x,h)] \ge E_{Q(h|x)}[\log P(x,h)]\\ &\Rightarrow \log \hat{I} \ge E_{Q(h|x)}[\log P(x,h)] \end{align} Here are two the puzzling parts:

1. I don't understand how they could drive

$$E_{Q(h|x)}[\log \hat{I}]\leq \log E_{Q(h|x)}[\hat{I}] = \log P(x)$$ given all mentioned in the above.

1. They also say since $\hat{I}$ is an unbiased, it can be written

$$E_{Q(h|x)}[ \hat{I}] = P(x)$$ which is not clear why, given the unbiased estimator definition.

Let's start with the second question first: The fact that $\hat{I}$ is unbiased can be seen as follows. First, in the paper it is also stated that the samples $h^i$ of the latent variables are independent. Thus, we have \begin{align} E_{Q(h|x)}(\hat{I}) & = E_{Q(h|x)}\left( \frac{1}{K} \sum_{i=1,\ldots,K} \frac{P(x,h)}{Q(h|x)}\right)\\ & = \frac{1}{K} \sum_{i=1,\ldots,K} E_{Q(h|x)}\left( \frac{P(x,h)}{Q(h|x)}\right)\\ & = \frac{1}{K} \sum_{i=1,\ldots,K} \int \frac{P(x,h)}{Q(h|x)} \cdot Q(h|x)\,\text{d}h\\ &= P(x) \end{align} The reason why this expectation is taken over $Q(h|x)$ rather than $P(x)$ is that we actually want to estimate the function $P(x)$. By taking the expectation $E_{P(x)}$, we would integrate out $x$ and obtain an expression that only depends on $h$ rather than $x$.
From this we can now easily obtain the answer to your first question. As $\log$ is concave, the inequality follows from Jensen's inequality which states that for concave functions $\phi(\cdot)$ we have $E(\phi(x)) \leq \phi(E(x))$. The equality follows from the proof above.
• given that $\hat{I}$ is estimating $P(x)$, shouldn't the unbiasness of $\hat{I}$ be proved by using $E_{P(x)}$ as opposed to $E_{Q(h|x)}$? this is a key thing I am not understanding. Aug 4, 2016 at 8:14
• so what is sufficient condition to prove an estimator is an unbiased estimator of a quantity? I thought that to prove $\hat{I}$ is unbiased estimator of $P(x)$, the expectation must be with respect to $P(x)$. Aug 4, 2016 at 11:18
• Your're right when estimating parameters $\theta$ of a parametric distribution $P_\theta(z)$ (e.g. the mean). Then proving unbiasedness of an estimator $\hat{\theta}(z_1,\ldots,z_n)$ from observations $z_i$ is carried out by checking $E_{P_\theta(z)}(\hat{\theta}(z_1, \ldots,z_n))=\theta$. In your question we do not merely estimate parameters of $P(x)$ but the function $P(x)$ itself from some "observations" $h_i$ (that is $h_i$ now play the role of $z_i$, $P(x)$ is the counterpart to $\theta$, $Q(h|x)$ is the counterpart of $P_\theta(z)$, and $\hat{I}$ is the counterpart to $\hat{\theta}$).