I am reading this ICML2016 paper, and am puzzled with the first inequality (converted to equality) on section 2.2.

Assume the model is $P(x,h)$ where $h$ are the hidden variables. Also assume $\hat{I}$ is an unbiased estimator of the likelihood term, $P(x)$. From this we can conclude that $$E_{P(x)}[\hat{I}]-P(x)=0\Rightarrow E_{P(x)}[\hat{I}] = P(x)$$ now assume we want to establish a lower bound on $P(x)$ (similar to EM approach) and plug in the estimator $\hat{I}$, instead of $P_{\theta}(x)$ in the lower bound formulation.

For this imagine the posterior distribution over the latent variables $h$, to be estimated using $Q(h|x)$ (i.e., $Q$ is a variational posterior). So if we want to write

\begin{align} &\log P(x) =\log \sum P(x,h)\\&\Rightarrow \log P(x) = \log \sum P(x,h)\frac{Q(h|x)}{Q(h|x)}\\&\Rightarrow \log E_{Q(h|x)}[P(x,h)] \ge E_{Q(h|x)}[\log P(x,h)]\\ &\Rightarrow \log \hat{I} \ge E_{Q(h|x)}[\log P(x,h)] \end{align} Here are two the puzzling parts:

  1. I don't understand how they could drive

$$E_{Q(h|x)}[\log \hat{I}]\leq \log E_{Q(h|x)}[\hat{I}] = \log P(x)$$ given all mentioned in the above.

  1. They also say since $\hat{I}$ is an unbiased, it can be written

$$E_{Q(h|x)}[ \hat{I}] = P(x)$$ which is not clear why, given the unbiased estimator definition.


1 Answer 1


Let's start with the second question first: The fact that $\hat{I}$ is unbiased can be seen as follows. First, in the paper it is also stated that the samples $h^i$ of the latent variables are independent. Thus, we have \begin{align} E_{Q(h|x)}(\hat{I}) & = E_{Q(h|x)}\left( \frac{1}{K} \sum_{i=1,\ldots,K} \frac{P(x,h)}{Q(h|x)}\right)\\ & = \frac{1}{K} \sum_{i=1,\ldots,K} E_{Q(h|x)}\left( \frac{P(x,h)}{Q(h|x)}\right)\\ & = \frac{1}{K} \sum_{i=1,\ldots,K} \int \frac{P(x,h)}{Q(h|x)} \cdot Q(h|x)\,\text{d}h\\ &= P(x) \end{align} The reason why this expectation is taken over $Q(h|x)$ rather than $P(x)$ is that we actually want to estimate the function $P(x)$. By taking the expectation $E_{P(x)}$, we would integrate out $x$ and obtain an expression that only depends on $h$ rather than $x$.

From this we can now easily obtain the answer to your first question. As $\log$ is concave, the inequality follows from Jensen's inequality which states that for concave functions $\phi(\cdot)$ we have $E(\phi(x)) \leq \phi(E(x))$. The equality follows from the proof above.

  • $\begingroup$ given that $\hat{I}$ is estimating $P(x)$, shouldn't the unbiasness of $\hat{I}$ be proved by using $E_{P(x)}$ as opposed to $E_{Q(h|x)}$? this is a key thing I am not understanding. $\endgroup$ Aug 4, 2016 at 8:14
  • $\begingroup$ Added an explanation of this to my original answer. $\endgroup$
    – Igor
    Aug 4, 2016 at 10:36
  • $\begingroup$ so what is sufficient condition to prove an estimator is an unbiased estimator of a quantity? I thought that to prove $\hat{I}$ is unbiased estimator of $P(x)$, the expectation must be with respect to $P(x)$. $\endgroup$ Aug 4, 2016 at 11:18
  • $\begingroup$ Your're right when estimating parameters $\theta$ of a parametric distribution $P_\theta(z)$ (e.g. the mean). Then proving unbiasedness of an estimator $\hat{\theta}(z_1,\ldots,z_n)$ from observations $z_i$ is carried out by checking $E_{P_\theta(z)}(\hat{\theta}(z_1, \ldots,z_n))=\theta$. In your question we do not merely estimate parameters of $P(x)$ but the function $P(x)$ itself from some "observations" $h_i$ (that is $h_i$ now play the role of $z_i$, $P(x)$ is the counterpart to $\theta$, $Q(h|x)$ is the counterpart of $P_\theta(z)$, and $\hat{I}$ is the counterpart to $\hat{\theta}$). $\endgroup$
    – Igor
    Aug 4, 2016 at 11:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.