I read information about variational autoencoder from this post: https://blog.keras.io/building-autoencoders-in-keras.html. To get reconstructed output, a value is sampled from Gaussian distribution in latent space, does anyone know why we need to use sampled data instead of the mean value itself to reconstruct the output? Would that be less accurate (since it introduces noise by sampling)?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
5
According to Auto-Encoding Variational Bayes (eq. 3), the loss function of a variational autoencoder
The expectation term is usually an intractable integral, so we want to approximate this expected value by drawing samples then computing the average.
The random value is added for generating samples from $q_\phi(z|x^{(i)})$, theoretically we should draw a large number of random values for an accurate approximation, but since the training usually takes thousands of iterations, we can use only one random value per input instance.
On the other hand, if we use the mean value instead, it'll no longer be the same loss function, as $$E[f(x)]\neq f(E[x]).$$
Update
In the original autoencoder models the data likelihood $p_\theta(x)$ as a whole so it's convenient to use mean square error or binary cross entropy to give a likelihood to optimize.
In the variational autoencoder we introduced the latent variable $z$, and want to model $q_\phi(z|x)$ (encoder) and $p_\theta(x|z)$ (decoder) jointly. The difficulty in this case is $p(x)$ can no longer be simply computed through one pass of the network as in the original autoencoder.
The paper addresses this by maximizing a variational lower bound of $p(x)$ (the above loss function), which contains an intractable expectation term which needs to be approximated using sampling and the "reparameterization trick".
I just found a very good Variational Autoencoder course on coursera, it mentions exactly your question in the forth video.
-
$\begingroup$ If I understand it correctly, the expectation term denotes reconstruction error, why do we need to use expectation representation (which requires sampling) instead of simply writing this term as deterministic mean squared error (so that no sampling is needed)? $\endgroup$ Nov 6, 2017 at 15:08
-
$\begingroup$ I did the derivation one more time, and not sure if this statement is correct: if we use deterministic value in latent space, then q_\phi becomes a delta function, and there is no way to optimize KL convergence between a delta function and another function p_\theta. The reason why they assume q_\phi to be Gaussian is that it is much easier to optimize KL term since it can be well written in closed form in terms of mean and variance of q_\phi. Is that the reason why we need to use probabilistic model instead of deterministic model in this case? $\endgroup$ Nov 6, 2017 at 16:32
-
1$\begingroup$ @username123 hi i've added some updates about the loss function. I'm not sure but I think you're right, if we chose q to be a delta function, then it'll be somehow the same as the original autoencoder and no sampling is required. But I guess the power this variational approach is allowing q to be a variety of distributions other than delta. $\endgroup$– dontlooNov 8, 2017 at 4:43
-
1$\begingroup$ @username123 hi I just found a very good Variational Autoencoder course on coursera, it mentions exactly your question in the forth video, coursera.org/learn/bayesian-methods-in-machine-learning/lecture/… $\endgroup$– dontlooDec 26, 2017 at 11:31
-