# Why maximizing the lower bound of variational evidence maximizes the probability of observing data

In vaiational bayesian inference we attempt to find a proxy function to best estimate the intractable posterior $P(z|X)$. We define best as the probability distribution that minimizes the KL divergence.

The final equation reached is:

$$log(P(X)) = KL(q_{\lambda}(z|X) || p(z|X)) + ELBO(\lambda)$$ where $log(P(X))$ is the log probability of observing the data(evidence)

$ELBO$ is the Evidence lower bound

$\lambda$ is the parameter over which the entire family of $q$ is taken.

Why does maximizing $ELBO(\lambda)$ maximizes the $log(P(X))$ as is claimed by prof. Ali Ghosdi here at 32:50

It would occur to me that maximizing $ELBO(\lambda)$ minimizes the $KL(q_{\lambda}(z|X) || p(z|X))$ and $log(P(X))$ is constant

The maximization of model evidence (observation) and the approximation of a posterior distribution is achieved via the Expectation Maximization algorithm (specifically the maximization-maximization variant).

This algorithm, then, has two steps:

Given that the posterior distribution is defined by $$\theta$$,

1. Compute a distribution $$q$$ over the range of $$z$$ such that $$q^{(t)}(z) = argmin_{q}KL(q(z)||p(z|X; \theta^{(t-1)}))$$ by maximising $$ELBO$$ w.r.t $$q(z)$$, while keeping $$\theta$$ fixed. For the sake of the theory behind this let's assume this ends up with $$q = p$$
This step is the the approximation of the posterior, which can be done in a variety of ways depending on the problem case and the distribution family of $$q$$. In this step, as you mentioned, $$log(p(X;\theta^{(t-1)}))$$ remains constant as it does not depend on $$q$$ and the KL-divergence linearly decreases as ELBO increases.

2. Now given that $$q^{(t)}(z) = argmin_{q}KL(q(z)||p(z|X; \theta^{(t-1)}))$$ maximize $$ELBO$$ w.r.t $$\theta$$, while keeping $$q(z)$$ fixed.
In this step, the KL-divergence potentially increases as ELBO increases, as $$KL(q(z)||p(z|X; \theta^{(t)})) \geq KL(q(z)||p(z|X; \theta^{(t-1)}))$$. Since $$KL(q(z)||p(z|X; \theta)) = -ELBO + log(p(X;\theta))$$, the only way this relationship holds is if $$log(p(X;\theta))$$ increases at least linearly as $$ELBO$$ increases w.r.t $$\theta$$

Once a maxima of $$log(p(X;\theta))$$ has been found, repeat step 1 to compute the best posterior approximation.