Understanding the Evidence Lower Bound (ELBO)

Question

I am reading this tutorial about Variational Inference, which includes the following depiction of ELBO as the lower bound on log-likelihood on the third page. In the tutorial, $x_i$ is the observed data and $z_i$ is the latent variable.

Let me first ask a quick clarification question. ELBO is defined in equation (4) as $\mathscr{L}(q, \theta) := -\operatorname*{KL}(q(z_i)||\mathbb{P}(x_i, z_i|\theta)).$ Since the KL divergence is non-negative, the ELBO must be non-positive. Am I right? The illustration seems to indicate that ELBO can be positive, so I am a little confused.

Now the main question: I understand ELBO is a lower bound of the log-likelihood so we want to make it as large as possible. However, why can we ignore the other term, $\operatorname*{KL}(q(z_i)||\mathbb{P}(z_i | x_i, \theta))$? In other words, how do we ensure increasing ELBO doesn't accidentally decrease $\operatorname*{KL}(q(z_i)||\mathbb{P}(z_i | x_i, \theta))$, given that they share the same $\theta$?

Answer 1

Don’t forget that log probabilities are non-positive, too. They’re the logs of values that cannot exceed 1.

With that in mind, the ELBO can be a meaningful lower bound on the log-likelihood: both are negative, but ELBO is lower. How much lower? The KL divergence from the conditional distribution.

I don’t see where you think the figure is indicating that it should be positive. The bottom of the diagram isn’t 0.

---

As for the second half of your question, see here: Should the likelihood function be increasing in every step of the EM algorithm?.

Answer 2

For the first question:

You are correct. The log likelihood is always nonpositive, and the ELBO is a lower bound on it, so is also certainly always nonpositive. That picture (and similar the one in Bishop 2006, §9.4, Figure 9.11) is rather misleading in my opinion, since it seems too imply that both of these quantities are positive. That cannot be. The bottom line in the picture is not labeled, and cannot be zero (in fact that bottom line and the indicated distances from it to the other lines doesn't represent anything meaningful as far as I can tell, which is why I think this picture is quite misleading).

To clarify, we have the following inequalities:

• First note that $\log \mathbb{P}(x\mid\theta) \le 0$ (the log likelihood is the log of a probability so it nonpositive by definition)

• Also, $\operatorname{KL}(\,q(z)\,\|\,\mathbb{P}(z\mid x,\theta)\,)\ge 0$ (since KL is always nonnegative)

• So,

  $$ \begin{aligned} \mathcal{L}(q,\theta) :=&~ \int\,q(z)\,\log\frac{\mathbb{P}(x,z\mid\theta)}{q(z)}\mathrm{d}z \\ =&~ \underbrace{\log \mathbb{P}(x\mid\theta)}_{\le 0} - \underbrace{\operatorname{KL}(\,q(z)\,\|\,\mathbb{P}(z\mid x,\theta)\,)}_{\ge 0} \le 0 \end{aligned} $$ since subtracting the KL from the log likelihood can only make the result more negative.

IMO, a less confusing version of the picture you shared would be something like the following:

In this picture, the horizontal lines represent real values. The dashed line is zero, the likelihood $\log \mathbb{P}(x\mid\theta)$ is some negative real value, and the ELBO $\mathcal{L}(\theta,q)$ is below that (more negative). The (positive) difference between these two lines is the $\operatorname{KL}(\,q(z)\,\|\,\mathbb{P}(z\mid x,\theta)\,)$.

I hope that clarifies somewhat.

---

For your second question as asking "how do we ensure increasing ELBO doesn't accidentally decrease the KL?": Look at the following equation:

$$ \operatorname{KL}(\,q(z)\,\|\,\mathbb{P}(z\mid x,\theta)\,) = \log \mathbb{P}(x\mid\theta)-\mathcal{L}(q,\theta) $$

Looking at it this way, the KL is broken down into two pieces, the likelihood and the ELBO. If you are looking to find the $q$ which maximizes this lowerbound for a fixed $\theta$, maximizing the ELBO by finding the optimal $q$ will necessarily only make the KL term smaller (making it vanish to zero, in the best case).

In a setting where you're doing a two step EM algorithm to get a maximum likelihood estimate, this maximization with respect to $q$ (while fixing $\theta$) is the E step. Then, in the subsequent M step, the distribution $q$ is held constant, and you optimize the ELBO with respect to $\theta$. Any increase in the ELBO that results will necessarily also increase the log likelihood term. Say you were at the optimal situation after the E step, and the KL term was zero. Since $q$ is held fixed during the M step that you just carried out, you will also have increased the KL to some nonzero positive number. That implies that the increase in the ELBO was smaller than increase the log likelihood.

See Bishop (2006 CH9) for further discussion of this EM algorithm.

I'd also highly recommend the discussion of this math in D. Kingma's thesis (2017 see section 2.3).

Answer 3

Regular expectation maximization works because the lower bound on the log likelihood is tight, so the M step will push up on the log-likelihood itself.

In the case of variational EM/inference, it is not the case that the lower bound is tight. Therefore, maximizing the lower bound can actually lead to a decrease in the actual log likelihood.

In terms of your question, you misread the equations. The definition of ELBO in the source you reference (eq 4) is

$$ \begin{aligned} \mathcal{L}(\theta,q) :=&~-\operatorname{KL}(q(z)\|\mathbb P(z, x|\theta))\\ =&~\log \mathbb P(x|\theta)-\operatorname{KL}(q(z)\|\mathbb P(z|x,\theta)) \end{aligned} $$

In the M step we maximize the left two expressions with respect to $\theta$. The rightmost expression containing the expression in your question is automatically maximized by maximizing the overall expression, so it is not being "ignored." Nonetheless, be careful that variational EM is not a tight lower bound so can lead to not finding the local, let alone global, maximum.