In the past month I've spent most of my time digging deep into how neural networks work, from the basic idea (to estimate the true posterior $p(z|x)$, we create a variational posterior $q(z|x)$ - the encoder), through the idea of maximizing ELBO minimizes the KL divergence between them, to making its breakdown in the way that we know all parts of it ($E_q[log \space p(x|z)] - KL(q(z|x)||p(z)$).
So the initial goal is to minimize the difference between how $q$ estimates $z$ given $x$, and how $p$ would do the same (if it would be tractable). That I don't completely understand why it works, but this belongs to a different question.
However, it does work, and given the breakdown, and how we calculate it in practice makes sense - cross entropy between $x$, and $p(x|z)$ minus difference between $q(z|x)$, and $p(z)$. In other words, as I understand it, force the network to reconstruct accurately, and make the latent distributions as close to a standard normal distributions as close as possible.
What I can't figure out is how this results in the latent distributions being continuous, so similar inputs encoded into latent variables end up next to each other, creating a smooth transition to different ones. Basically the end difference between a variational, and deterministic autoencoder. In other words, our loss function cares about the encoder outputting distributions close to each other, and the decoder outputting similar outputs to the inputs, and I can't see a reason why the network couldn't "cheat" acting like a standard encoder. Given two similar inputs, if it's easier to reconstruct for the network, it could encode them into latent distributions far from each other, but still similar to a standard distributions. Both parts of the loss function are satisfied, but they won't be continuous, so close to each other.
What piece of information am I missing here? How could I think about it in a way that I can imagine, and would make sense?
