8
$\begingroup$

From my understanding of VAE's, there's a step during training in the middle where, after the encoder produces a mean and standard deviation, random samples are drawn from the given learned distribution to create the encoded vector that the decoder works to decode. I understand how one uses the KL divergence to force the learned distribution to be approximately the standard Gaussian, but I don't understand how the reconstruction loss can be back propagated past this sampling step. Random sampling is not a differentiable operation, so how can the gradients propagate past it? Is my understanding of VAE's wrong?

$\endgroup$
10
$\begingroup$

The reparameterization trick.

$$x = \text{sample}(\mathcal{N}(\mu, \sigma^2))$$

is not backpropable wrt $\mu$ or $\sigma$. However, we can rewrite this as:

$$x = \mu + \sigma\ \text{sample}( \mathcal{N}(0, 1))$$

which is clearly equivalent and backpropable.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Does this mean that we can't build autoencoders that use a different distribution that can't be reparameterized this way? $\endgroup$ – enumaris Apr 26 '18 at 15:52
  • 1
    $\begingroup$ @enumaris most distributions can be reparameterized. For example, you can use a categorical latent space using the gumbel softmax trick. $\endgroup$ – shimao Apr 26 '18 at 15:55
  • 2
    $\begingroup$ But in theory the normal distribution is all you'll ever need, since a sufficiently powerful function approximator can always map the normal distribution to any arbitrary probability distribution. $\endgroup$ – shimao Apr 26 '18 at 15:56
  • 1
    $\begingroup$ @blue-phoenix Scaling a random variable by a factor of $k$ scales the variance by a factor of $k^2$ $\endgroup$ – shimao Aug 9 '18 at 7:40
  • 1
    $\begingroup$ @blue-phoenix there's no need to add the square root. The fact that a particular implementation of the normal distribution is parameterized by the standard deviation rather than the variance doesn't change any of the math. $\endgroup$ – shimao Aug 9 '18 at 7:58

Not the answer you're looking for? Browse other questions tagged or ask your own question.