6
$\begingroup$

From my understanding of VAE's, there's a step during training in the middle where, after the encoder produces a mean and standard deviation, random samples are drawn from the given learned distribution to create the encoded vector that the decoder works to decode. I understand how one uses the KL divergence to force the learned distribution to be approximately the standard Gaussian, but I don't understand how the reconstruction loss can be back propagated past this sampling step. Random sampling is not a differentiable operation, so how can the gradients propagate past it? Is my understanding of VAE's wrong?

$\endgroup$
8
$\begingroup$

The reparameterization trick.

$$x = \text{sample}(\mathcal{N}(\mu, \sigma^2))$$

is not backpropable wrt $\mu$ or $\sigma$. However, we can rewrite this as:

$$x = \mu + \sigma\ \text{sample}( \mathcal{N}(0, 1))$$

which is clearly equivalent and backpropable.

$\endgroup$
  • 1
    $\begingroup$ Does this mean that we can't build autoencoders that use a different distribution that can't be reparameterized this way? $\endgroup$ – enumaris Apr 26 '18 at 15:52
  • 1
    $\begingroup$ @enumaris most distributions can be reparameterized. For example, you can use a categorical latent space using the gumbel softmax trick. $\endgroup$ – shimao Apr 26 '18 at 15:55
  • 2
    $\begingroup$ But in theory the normal distribution is all you'll ever need, since a sufficiently powerful function approximator can always map the normal distribution to any arbitrary probability distribution. $\endgroup$ – shimao Apr 26 '18 at 15:56
  • 1
    $\begingroup$ @blue-phoenix Scaling a random variable by a factor of $k$ scales the variance by a factor of $k^2$ $\endgroup$ – shimao Aug 9 '18 at 7:40
  • 1
    $\begingroup$ @blue-phoenix there's no need to add the square root. The fact that a particular implementation of the normal distribution is parameterized by the standard deviation rather than the variance doesn't change any of the math. $\endgroup$ – shimao Aug 9 '18 at 7:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.