In Goodfellow's Generative Adversarial Nets, it is mentioned that

Our work backpropagates derivatives through generative processes by using the observation that $$\lim_{\sigma \rightarrow 0} \nabla_x \mathbb E_{\epsilon \sim N(0,σ^2 \mathbf I)} f (x + \epsilon) = \nabla_x f (x)$$

What exactly does this mean, and how is it of relevance to the way backpropagation is performed? This is mentioned at the beginning of the paper and is not referred to again, so it's not clear to me why this is significant (or how it affects the learning process).

Right after this, it is said that

We were unaware at the time we developed this work that Kingma and Welling [18] and Rezende et al. [23] had developed more general stochastic backpropagation rules, allowing one to backpropagate through Gaussian distributions with finite variance, and to backpropagate to the covariance parameter as well as the mean. These backpropagation rules could allow one to learn the conditional variance of the generator, which we treated as a hyperparameter in this work.

So what exactly is different between these two implementations? What does Goodfellow mean by "backpropagating covariance/mean through Gaussian distributions" and "conditional variance of the generator"?


In both a GAN model and a VAE model, there is a decoder network $g$ (in GAN literature, the decoder is usually called the generator instead).

In both cases, we define a latent variable $z$ to be distributed normally, usually $z \sim \mathcal{N}(0, I)$. A typical GAN or VAE model defines a density on images:

$$p(x) = \int p(z)p(x|z) dz $$ where $p(x|z)$ is gaussian, with mean and variance determined by the generator network: $\mu, \sigma = g(z; \theta)$. In the case of this GAN work, only $\mu$ was determined by $g$, and $\sigma^2$, the "conditional variance of the generator", was fixed as a hyperparameter.

You can sample an image $x$ by first sampling some $z$ from the standard normal prior, then sample $x$ from $\mathcal{N}(\mu,\sigma=g(z;\theta))$. In both the case of VAE and GAN, you'll eventually want to backpropagate some function of $x$, and compute a value such as $\nabla_\theta f(x)$.

Notation overload warning: what is "$x$" in the equation in OP is $\mu$ here, and what is $x$ here is "$x + \epsilon$" in OP.

This is problematic because sampling isn't a function you can differentiate through, so it's not clear how to compute the gradient. One approach taken by Kingma / Welling / Rezende is to "reparameterize" $x$ as $x = \mu + \sigma \epsilon$ where $\epsilon$ is a distributed as a standard normal. Basically this shifts the sampling operation over to $\epsilon$, so that it doesn't "get in the way" of differentiation. Of course the resulting $x$ is still drawn from the same distribution as before. Note that you can backpropagate through both $\mu$ and $\sigma$ (which are functions of $\theta$) now.

Goodfellow had a different solution to this problem: just substitute $\nabla_\theta f(x)$ with $\nabla_\theta f(\mu)$. Equation 1 states that in the limit, the expectation of the former is the latter.

  • $\begingroup$ Note that in a GAN, the conditional variance $\sigma^2$ is (almost) always just $0$. You can motivate things that way, via equation 1, but you don't really need to, and later explanations of GANs (including Goodfellow's 2016 tutorial) don't do that. $\endgroup$
    – Danica
    Nov 15 '19 at 6:05
  • $\begingroup$ Thanks! Would you happen to know of any resources comparing performance of vanilla versus reparametrized GANs? $\endgroup$
    – actinidia
    Nov 19 '19 at 5:54
  • $\begingroup$ I don't think anyone has actually bothered to try it out... as Dougal mentions, most people leave the variance at 0. $\endgroup$
    – shimao
    Nov 19 '19 at 6:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.