Why maximizing the lower bound of variational evidence maximizes the probability of observing data

Question

In vaiational bayesian inference we attempt to find a proxy function to best estimate the intractable posterior $P(z|X)$. We define best as the probability distribution that minimizes the KL divergence.

The final equation reached is:

$$log(P(X)) = KL(q_{\lambda}(z|X) || p(z|X)) + ELBO(\lambda)$$ where $log(P(X))$ is the log probability of observing the data(evidence)

$ELBO$ is the Evidence lower bound

$\lambda$ is the parameter over which the entire family of $q$ is taken.

Why does maximizing $ ELBO(\lambda)$ maximizes the $log(P(X))$ as is claimed by prof. Ali Ghosdi here at 32:50

It would occur to me that maximizing $ ELBO(\lambda)$ minimizes the $KL(q_{\lambda}(z|X) || p(z|X))$ and $log(P(X))$ is constant

Answer 1

The maximization of model evidence (observation) and the approximation of a posterior distribution is achieved via the Expectation Maximization algorithm (specifically the maximization-maximization variant).

This algorithm, then, has two steps:

Given that the posterior distribution is defined by $\theta$,

1. Compute a distribution $q$ over the range of $z$ such that $q^{(t)}(z) = argmin_{q}KL(q(z)||p(z|X; \theta^{(t-1)}))$ by maximising $ELBO$ w.r.t $q(z)$, while keeping $\theta$ fixed. For the sake of the theory behind this let's assume this ends up with $q = p$
   This step is the the approximation of the posterior, which can be done in a variety of ways depending on the problem case and the distribution family of $q$. In this step, as you mentioned, $log(p(X;\theta^{(t-1)}))$ remains constant as it does not depend on $q$ and the KL-divergence linearly decreases as ELBO increases.

2. Now given that $q^{(t)}(z) = argmin_{q}KL(q(z)||p(z|X; \theta^{(t-1)}))$ maximize $ELBO$ w.r.t $\theta$, while keeping $q(z)$ fixed.
   In this step, the KL-divergence potentially increases as ELBO increases, as $KL(q(z)||p(z|X; \theta^{(t)})) \geq KL(q(z)||p(z|X; \theta^{(t-1)}))$. Since $KL(q(z)||p(z|X; \theta)) = -ELBO + log(p(X;\theta))$, the only way this relationship holds is if $log(p(X;\theta))$ increases at least linearly as $ELBO$ increases w.r.t $\theta$

Once a maxima of $log(p(X;\theta))$ has been found, repeat step 1 to compute the best posterior approximation.