VQ-VAE objective - is it ELBO maximization, or minimization of the KL-divergence between the posterior and its approximation? I'm reading two descriptions of the VQ-VAE objective:
Kingma claims in page 18 that we want to maximize the ELBO, and shows that it can be written as $ELBO = logp_{\theta}(x) - KL(q_{\phi}(z|x)||p_{\theta}(z|x))$, the marginal likelihood of the data - the KL divergence between the approximate posterior and the true posterior of the latent variables.
Rocca claims that we just want to minimize the same $KL(q_{\phi}(z|x)||p_{\theta}(z|x))$ Kingma mentioned, but doesn't mention ELBO or the marginal likelihood of the data.
Are they saying the same thing? if so, why? is it because the marginal likelihood of the data is not a function of our variational parameters $\phi$? which is the "true" objective and which is just a math development? I need a good intuition on what is it that we're optimizing here...
 A: Let's consider
$$
\begin{align}
\left({\phi}^{*},{\theta}^{*}\right) &\in
\operatorname*{arg\,max}_{\phi,\,\theta} \, \operatorname{ELBO}\left(\phi,\theta\right)  \\&= 
\operatorname*{arg\,min}_{\phi,\,\theta} \, \left\{-\operatorname{ELBO}\left(\phi,\theta\right)\right\} \\&=  
\operatorname*{arg\,min}_{\phi,\,\theta} \, \left\{\operatorname{KL}\left(q_\phi\left(z\right)\|\ p_\theta\left(z|x\right)\right) - \ln\left(p_\theta\left(x\right)\right)\right\}  
\end{align}
$$
for optimal parameters ${\theta}^{*}$ and ${\phi}^{*}$ with an asscociated optimal variational density $q_{\phi^*}\!\left(z\right)$.
In many Bayesian settings, where $\theta \equiv z$ is considered as random vector, an approximation to the posterior density $p_\theta\left(z|x\right) \equiv p\left(\theta|x\right)$ is sought and $p_\theta\left(x,z\right) \equiv p\left(x,\theta\right)$ is the product of the prior density $p_\theta\left(z\right) \equiv p\left(\theta\right)$ and the density of the observation model/likelihood $p_\theta\left(x|z\right) \equiv p\left(x|\theta\right)$. In this case, the ELBO and KL-divergence become independent of $\theta$ and maximizing $\operatorname{ELBO}\left(\phi\right)$ w.r.t. $\phi$ is equivalent to minimizing the KL-divergence w.r.t. $\phi$ since $p_\theta\left(x\right)$ is by assumption constant w.r.t. $\phi$. The KL-divergence is a common measure of discrepancy between two probabilit densities (but not a metric) which is non-negative and attains zero iff they coincide almost everywhere. One idea could thus be to find an "optimal" approximation $q_{\phi^*}\!\left(\theta\right)$ to the posterior density by minimizing the KL-divergence between $q_\phi\left(\theta\right)$ and $p\left(\theta|x\right)$ w.r.t. $\phi$. However, minimizing the KL-divergence directly is assumed to be inadmissible as it depends on the true posterior density, which is unknown or intractable (otherwise there would be no need to find an approximation to it in the first place). But in the evidence lower bound $\operatorname{ELBO}\left(\phi\right) \equiv \int \ln\left(p\left(x,\theta\right)/q_\phi\left(\theta\right)\right)q_\phi\left(\theta\right)\mathrm d \theta$, we are no longer dealing with $p\left(\theta|x\right)$ but with the joint density $p\left(x,\theta\right)$ which is assumed to have a known functional form.
In different settings, when dealing with models where the likelihood $p_\theta\left(x|z\right)$ is conditioned on a latent vector $z$, the "ELBO" is a log-likelihood lower bound. Now, by maximizing this lower bound, given by $\operatorname{ELBO}\left(\phi,\theta\right) \equiv \int \ln\left(p_\theta\left(x,z\right)/q_\phi\left(z\right)\right)q_\phi\left(z\right)\mathrm d z$, over $\phi$ and $\theta$, the goal is to both minimize the KL-divergence between $q_\phi\left(z\right)$ and $p_\theta\left(z|x\right)$ and maximize the approximate log-likelihood. Here, ${\theta}^{*}$ is a variational approximation to the MLE.

Reference
Ormerod, J. T., & Wand, M. P. (2010). Explaining variational approximations. The American Statistician, 64(2), 140-153.
