I've been spending almost a month trying to understand VAE. I was reading a bunch of tutorials, and at first, it made sense and seemed straightforward. Then I was experimenting with it, it produced weird results, had no idea what was wrong. While trying to figure it out realised that I don't understand it at all.
This question will consist of multiple small questions that I probably should ask separately, but it would lose the context, so I will try to separate them.
My first impression was that the only difference between AE and VAE is:
- the encoder, instead of producing a single point in N-dimensional space, is producing an N-dimensional range where the point can be;
- the KL divergence loss - added to the reconstruction loss - is somehow magically forcing the encoder to produce the ranges in the way that they will overlap ("touch") each other, and be centred, so the ranges will try to be close to 0 as possible, but at the same time ranges (distributions) with different classes will push each other.
When I just simply looked at the equation (of KL loss), it sort of made sense (however, didn't understand why it did) why it does centring the distributions, and why it made the distributions overlap, but didn't (and doesn't) make sense at all how it forces the distributions of same classes to "group" with each other. Simply because the KL loss equation (seemingly) has no idea about which class is which, and the reconstruction loss equation doesn't care how far away the same classes are from each other (and has no direct connection with the latent distribution of the given observation) as long as the observations are close to the expected output.
Then I realised that classes in VAE don't even exist, it is just in my diagram rendering the means given the MNIST dataset I used for training, and colored the points with the corresponding classes.
So the encoder is trying to organise the inputs to put similar ones close to each other, and slowly transform them into different inputs. But still not understanding how.
I started to learn about KL divergence, and how it works. KL divergence (or relative entropy) compares probability distributions and measures how different they are from each other. The question I was trying to find the answer for is in terms of VAE, what are we comparing with what?
After reading, seeing examples, and experimenting, I thought we're comparing the distributions on the batch axis, that would make sense. It would try to make the distributions very similar to each other, but the reconstruction loss wouldn't let them, and the tension between them would produce the beautiful transition of the different outputs.
But then I realised that this isn't the case, as the equation doesn't directly make the different distributions to contact with each other. I found the brief explanation of the equation we're using in Wikipedia:
A special case, and a common quantity in variational inference, is the relative entropy between a diagonal multivariate normal, and a standard normal distribution (with zero mean and unit variance):
So I realised that we're comparing our distributions with the distribution where we sampled $\epsilon$ randomly, which is a normal distribution with mean zero and standard deviation equal to one.
Does this mean that we're just forcing our distributions to have 0 means, and standard deviations 1? Or if there are other goals, how do we earn them with this equation?
I see the elements of VAE mentioned as the terms of Bayes's theorem. For example, a very detailed, and nice tutorial mentions:
The weight of KL divergence in the loss function is a hyperparameter we shouldn’t ignore at all: it adjusts the “distance” between the prior and posterior distribution of z, and plays a decisive role in the performance of the model.
So as I read about the Bayes theorem, in short, the posterior shows the probability that our hypothesis is true given the evidence is true, by multiplying the likelihood (probability of the evidence is true given the hypothesis is true) with the prior (independent probability of the hypothesis is true), divided with the independent probability of evidence is true.
Given the examples that made me sort of understand the Bayes theorem, it is not too clear in VAEs, which part is the prior, posterior, likelihood, hypothesis, and evidence?