5
$\begingroup$

I'm trying to get my head around variational inference, but I'm confused with the definition of ELBO, specifically an expectation over joint distribution. Here I use Variational Inference: A Review for Statisticians as a reference, although other sources use similar notation.

In Eq. 13 authors introduce ELBO as:

$$ELBO(q) = \mathbb{E}[\log p(z, x)] - \mathbb{E}[\log q(z)]$$

where expectations are taken w.r.t. $q(z)$.

If my math doesn't lie to me, the second term is expanded into:

$$\mathbb{E}[\log q(z)] = \mathbb{E_{q(z)}}[\log q(z)] = \int q(z) \cdot \log q(z) \cdot dz$$

and is integrated into a single number. It looks good to me and works well as an optimization objective. However, the first term is:

$$\mathbb{E}[\log p(z, x)] = \mathbb{E_{q(z)}}[\log p(z, x)] = \int q(z) \cdot \log p(z,x) \cdot dz$$

I.e. it contains joint distribution $\log p(z, x)$. Even if we integrate out $z$, we still have a random variable $x$ with multiple possible values. So as I understand it, the whole term $\mathbb{E}[\log p(z, x)]$ is a probability density itself.

If both statements above are correct, I understand that the definition of ELBO now looks like:

$$ELBO(q) = <\text{some density over }x> - <\text{scalar}>$$

I believe this means substracting a constant from each value of density in the first term. It also plays well with Eq. 14 in the paper which explains definition for evidence lower bound itself.

However, if value of ELBO is a distribution, what at all does it mean to optimize it?

To summarize my questions:

  • is value of ELBO a single number or a distribution?
  • if a number, how can be a lower bound of $\log p(x)$?
  • if a distribution, how do we optimize it?
  • are there any other mistakes in my understanding?
Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ You can think of the ELBO as a function from the the family of distributions you're optimizing over to $\mathbb{R}$. The reason it outputs a scalar is because you're assuming that you're working with some fixed $x$, so that $\log(p(x))$ is a constant and $p(z, x)$ is only a function in $z$. $\endgroup$
    – aleshing
    Jan 2, 2018 at 22:28
  • $\begingroup$ Working with a fixed $x$ is unusual for me, but it actually makes sense. I need to re-read the paper with this idea in mind, but meanwhile: in derivations, do we assume that this fixed $x$ is a single observation or the whole dataset that we should somehow summarize? In other words, is, for example, $p(z | x)$ a probability of $z$ given some concrete sample of $x$? $\endgroup$
    – ffriend
    Jan 3, 2018 at 7:12
  • $\begingroup$ It's fixed in that when you're actually implementing variational inference (or any other method of approximate inference) for some model, you're working with an actual data set. This data set is fixed, and when you calculate $p(x)$ for this data set, it will spit out some scalar. $p(z|x)$ is indeed the posterior probability of $z$, the unknowns, given your data $x$. How much have you studied Bayesian statistics before? It seems as though you might have a misunderstanding about the underlying framework that variational inference is used in (approximate bayesian inference). $\endgroup$
    – aleshing
    Jan 3, 2018 at 19:46
  • $\begingroup$ Indeed, I'm a software engineer with experience in (non-Bayesian) machine learning with some serious gaps in understanding of Bayesian statistics, so suggestions for a modern well-structured introduction to the topic is also welcome :) Anyway, thanks for an excellent explanation! $\endgroup$
    – ffriend
    Jan 3, 2018 at 22:39
  • 1
    $\begingroup$ People recommend Statistical Rethinking by McElreath as a good first pass introduction for building intuition. Books at a slightly higher level are Doing Bayesian Data Analysis by Kruschke and A First Course in Bayesian Statistical Methods by Hoff. The graduate level reference is Bayesian Data Analysis by Gelman et al. Bayesian Data Analysis is the only one that covers variational inference, but all of them should give you enough background to read through the the JASA review you're going through. $\endgroup$
    – aleshing
    Jan 3, 2018 at 23:32

1 Answer 1

Reset to default
3
$\begingroup$

Technically ELBO would be a functional, a function that takes a function as an argument. However, in practice most problems assume some class of distributions (e.g. Gaussian, Gamma, etc), which eliminates the functional aspect of the problem and then optimize within this class of distributions, making the problem a single variable, or multivariate problem depending on how many parameters are in the family.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Ah, I should have written "value of ELBO" instead of just "ELBO", i.e. what is the codomain of ELBO. I fixed it in text. But from what you say about functionals, I understand ELBO's value is in $\mathbb{R}$, i.e. $\mathbb{E}[\log p(z, x)]$ is a number. Is it correct? If so, why do we ignore that $x$ is a r.v. itself? $\endgroup$
    – ffriend
    Jan 3, 2018 at 7:05
  • $\begingroup$ You don't really ignroe that $x$ is a random variable, $x$ is the observed data. In most VB applications you want to estimate the parameters of a distribution using observed data. The $z$ is unobserved data and is therefore integrated/taken an expectation over $z$. What you notation obscures is that these distributions have parameter(s) that you want to estimate. $\endgroup$
    – Lucas Roberts
    Jan 3, 2018 at 14:19
  • $\begingroup$ @LucasRoberts $x$ is observed, but usually it's assumed that $x\sim P_\theta(x)$, some sampling distribution. Suppose we want to incorporate this randomness of $x$, then the original idea (of ELBO as a distribution) sounds interesting $\endgroup$
    – lady gaga
    Mar 15, 2021 at 18:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.