1
$\begingroup$

I'm trying to derive the ELBO as per CS229 Page 144, Eq 11.8.

For E-step, the LHS and RHS must be equal: $$ \begin{align} f\left(\mathbb{E}_{z\sim Q}\left[ \frac{p(x, z; \theta)}{Q(z)}\right]\right) \geq \mathbb{E}_{z\sim Q}\left[f\left( \frac{p(x, z; \theta)}{Q(z)}\right)\right] \end{align} $$

The value must be a constant random variable for equality to hold: $$ \begin{align} \frac{p(x, z; \theta)}{Q(z)} = c \end{align} $$

This can be accomplished by choosing $Q(z) \propto p(x, z; \theta)$. Also, $\sum_z Q(z) = 1$.

Part that is not clear: $$ \begin{align} Q(z) = \frac{p(x, z; \theta)}{\sum_z p(x, z; \theta)} \end{align} $$ I'm aware of the simplification that follows this equation. But how did they arrive at this? Which property am I missing?

Improve this question
$\endgroup$
3
  • $\begingroup$ Maybe I don't properly understand your question: You look for a function that differs from $p$ only by a factor and that is also normalized. So just normalize it, i.e. divide by the sum of all values. You can check that now both conditions are satisfied. $\endgroup$
    – frank
    Commented Sep 8, 2022 at 6:39
  • $\begingroup$ I get that the denominator is independent of $z$ after marginalization, and the whole fraction reduces to $p(z|x)$. My question is why that exact term in the denominator and nothing else? Most derivations directly assume $Q(z)=p(z|x)$, which is in fact derived from the equation above. $\endgroup$
    – Karan Shah
    Commented Sep 8, 2022 at 7:39
  • $\begingroup$ My question is a duplicate. Found the solution here $\endgroup$
    – Karan Shah
    Commented Sep 8, 2022 at 7:43

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.