Why do we start from $KLD(q(z) || p(z|x))$ in ELBO derivation?

Question

Of the several ways to derive the evidence lower bound (such as using Jensen’s inequality), a version often used is the derivation from $KLD(q(z) || p(z|x))$.

The following image illustrates the derivation process (screenshotted from here):

While I do understand how the math works out, I do not have an argument for using $KLD(q(z) || p(z|x))$ as a starting point. Why $KLD(q(z) || p(z|x))$, and not $KLD(q(z) || p(z))$ or $KLD(q(z|x) || p(z|x))$?

If someone asks me the motivation of wanting to calculate $KLD(q(z) || p(z|x))$ (as opposed to KLD between other distributions, as listed above), what can I tell them?

Answer 1

For Variational Bayes, the aim is to approximate the intractable posterior distribution $p(z \mid x)$ with a variational approximation $q(z)$ which is an easy, parametric distribution to perform posterior inference. That is why we want to optimize the KLD between $q(z)$ and $p(z|x)$. You can review Variational Inference: A Review for Statisticians section 2.2.

As in your derivations, it's most of the time impossible to directly optimize this KLD due to the $\log p(x)$ being intractable. This intractability because evidence integral is unavailable in closed form or requires exponential time to compute. Assuming that $p(x,z) = p(z) p(x \mid z)$, an important assumption in VI, you minimize the KLD between

TL;DR, VI tries to find a $q(z)$ closest to the $p(z \mid x)$ via minimizing the KL divergence between them. That's the original problem. Your derivations are showing the equivalent problem which is required due to the evidence term.

For more technical depth, it's possible to define VI over priors or even function spaces. Doubly Semi-Implicit Variational Inference can be good reading for defining VI over both posterior and prior.