1
$\begingroup$

In this tutorial, it has been explained why maximizing the ELBO minimizes KL divergence, but then this:

$$ELBO_\lambda = E_q[\log \space p(x,z)] - E_q[\log \space q_\lambda(z|x)]$$

became this:

$$ELBO_\lambda = E_{q_\lambda(z|x)}[\log \space p(x|z)] - KL(q_\lambda(z|x)||p(z))$$

and I don't understand how. What I also noticed, that while in the first equation, there are expected values with respect to $q$, in the lower index, but in the second, it is not in the lower index, so I don't know what $E_{q_\lambda(z|x)}[]$ is supposed to mean.

$\endgroup$
0

1 Answer 1

1
$\begingroup$

In slightly different notation, with $f(y, \theta)$ the joint density of parameter and data, $q(\theta)$ the approximating function, prior $\pi(\theta)$ and likelihood $f(y|\theta)$, we have $$ \text{ELBO}(q)=E_{q(\theta)}\left[\log f(y, \theta)\right] - E_{q(\theta)}\left[\log q(\theta) \right]$$ Expanding $f(y, \theta)$ gives $$ \begin{aligned} \text{ELBO}(q) &= \underbrace{E_{q(\theta)}\left[\log f(y|\theta)\right] + E_{q(\theta)}\left[\log \pi(\theta)\right]}_{E_{q(\theta)}\left[\log f(y, \theta)\right]} - E_{q(\theta)}\left[\log q(\theta) \right] \\ &= E_{q(\theta)}\left[\log f(y|\theta)\right] + E_{q(\theta)}\left[\log \frac{\pi(\theta)}{q(\theta)}\right] \\ &= E_{q(\theta)}\left[\log f(y|\theta)\right] - E_{q(\theta)}\left[\log \frac{q(\theta)}{\pi(\theta)}\right] \\ &= E_{q(\theta)}\left[\log f(y|\theta)\right] - \text{KL}\left(q(\theta) \, \lvert\lvert \, \pi(\theta)\right) . \end{aligned}$$ Here, indeed, the expectation is taken over the same distribution in either expression.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.