I'm trying to get my head around variational inference, but I'm confused with the definition of ELBO, specifically an expectation over joint distribution. Here I use Variational Inference: A Review for Statisticians as a reference, although other sources use similar notation.
In Eq. 13 authors introduce ELBO as:
$$ELBO(q) = \mathbb{E}[\log p(z, x)] - \mathbb{E}[\log q(z)]$$
where expectations are taken w.r.t. $q(z)$.
If my math doesn't lie to me, the second term is expanded into:
$$\mathbb{E}[\log q(z)] = \mathbb{E_{q(z)}}[\log q(z)] = \int q(z) \cdot \log q(z) \cdot dz$$
and is integrated into a single number. It looks good to me and works well as an optimization objective. However, the first term is:
$$\mathbb{E}[\log p(z, x)] = \mathbb{E_{q(z)}}[\log p(z, x)] = \int q(z) \cdot \log p(z,x) \cdot dz$$
I.e. it contains joint distribution $\log p(z, x)$. Even if we integrate out $z$, we still have a random variable $x$ with multiple possible values. So as I understand it, the whole term $\mathbb{E}[\log p(z, x)]$ is a probability density itself.
If both statements above are correct, I understand that the definition of ELBO now looks like:
$$ELBO(q) = <\text{some density over }x> - <\text{scalar}>$$
I believe this means substracting a constant from each value of density in the first term. It also plays well with Eq. 14 in the paper which explains definition for evidence lower bound itself.
However, if value of ELBO is a distribution, what at all does it mean to optimize it?
To summarize my questions:
- is value of ELBO a single number or a distribution?
- if a number, how can be a lower bound of $\log p(x)$?
- if a distribution, how do we optimize it?
- are there any other mistakes in my understanding?