1
$\begingroup$

I am reading this tutorial about Variational Inference, which includes the following depiction of ELBO as the lower bound on log-likelihood on the third page. In the tutorial, $x_i$ is the observed data and $z_i$ is the latent variable.

enter image description here

Let me first ask a quick clarification question. ELBO is defined in equation (4) as $\mathscr{L}(q, \theta) := -\operatorname*{KL}(q(z_i)||\mathbb{P}(x_i, z_i|\theta)).$ Since the KL divergence is non-negative, the ELBO must be non-positive. Am I right? The illustration seems to indicate that ELBO can be positive, so I am a little confused.

Now the main question: I understand ELBO is a lower bound of the log-likelihood so we want to make it as large as possible. However, why can we ignore the other term, $\operatorname*{KL}(q(z_i)||\mathbb{P}(z_i | x_i, \theta))$? In other words, how do we ensure increasing ELBO doesn't accidentally decrease $\operatorname*{KL}(q(z_i)||\mathbb{P}(z_i | x_i, \theta))$, given that they share the same $\theta$?

$\endgroup$

2 Answers 2

2
$\begingroup$

Don’t forget that log probabilities are non-positive, too. They’re the logs of values that cannot exceed 1.

With that in mind, the ELBO can be a meaningful lower bound on the log-likelihood: both are negative, but ELBO is lower. How much lower? The KL divergence from the conditional distribution.

I don’t see where you think the figure is indicating that it should be positive. The bottom of the diagram isn’t 0.


As for the second half of your question, see here: Should the likelihood function be increasing in every step of the EM algorithm?.

$\endgroup$
1
$\begingroup$

Regular expectation maximization works because the lower bound on the log likelihood is tight, so the M step will push up on the log-likelihood itself.

In the case of variational EM/inference, it is not the case that the lower bound is tight. Therefore, maximizing the lower bound can actually lead to a decrease in the actual log likelihood.

In terms of your question, you misread the equations. The definition is

$$L(\theta,q)=KL(q(z)\|\mathbb P(z, x|\theta))=\log \mathbb P(x|\theta)-KL(q(z)\|\mathbb P(z|x,\theta))$$

In the M step we maximize the left two expressions with respect to $\theta$. The rightmost expression containing the expression in your question is automatically maximized by maximizing the overall expression, so it is not being "ignored." Nonetheless, be careful that variational EM is not a tight lower bound so can lead to not finding the local, let alone global, maximum.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.