From my understanding of VAE's, there's a step during training in the middle where, after the encoder produces a mean and standard deviation, random samples are drawn from the given learned distribution to create the encoded vector that the decoder works to decode. I understand how one uses the KL divergence to force the learned distribution to be approximately the standard Gaussian, but I don't understand how the reconstruction loss can be back propagated past this sampling step. Random sampling is not a differentiable operation, so how can the gradients propagate past it? Is my understanding of VAE's wrong?
$\begingroup$
$\endgroup$
1
-
5$\begingroup$ Does this answer your question? How does the reparameterization trick for VAEs work and why is it important? $\endgroup$– David DaoCommented Jun 17, 2020 at 22:51
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
11
The reparameterization trick.
$$x = \text{sample}(\mathcal{N}(\mu, \sigma^2))$$
is not backpropable wrt $\mu$ or $\sigma$. However, we can rewrite this as:
$$x = \mu + \sigma\ \text{sample}( \mathcal{N}(0, 1))$$
which is clearly equivalent and backpropable.
-
1$\begingroup$ Does this mean that we can't build autoencoders that use a different distribution that can't be reparameterized this way? $\endgroup$– enumarisCommented Apr 26, 2018 at 15:52
-
1$\begingroup$ @enumaris most distributions can be reparameterized. For example, you can use a categorical latent space using the gumbel softmax trick. $\endgroup$– shimaoCommented Apr 26, 2018 at 15:55
-
3$\begingroup$ But in theory the normal distribution is all you'll ever need, since a sufficiently powerful function approximator can always map the normal distribution to any arbitrary probability distribution. $\endgroup$– shimaoCommented Apr 26, 2018 at 15:56
-
1$\begingroup$ @blue-phoenix Scaling a random variable by a factor of $k$ scales the variance by a factor of $k^2$ $\endgroup$– shimaoCommented Aug 9, 2018 at 7:40
-
1$\begingroup$ @blue-phoenix there's no need to add the square root. The fact that a particular implementation of the normal distribution is parameterized by the standard deviation rather than the variance doesn't change any of the math. $\endgroup$– shimaoCommented Aug 9, 2018 at 7:58