I came across this equation in the original GAN paper (pg 2 https://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf): $$\lim_{\sigma \rightarrow 0} \nabla_{\bf x} \mathbb{E}_{\epsilon \sim \mathcal{N}(0, \sigma^2 I)}[{f({\bf x} + \epsilon)}] = \nabla_{\bf x} f({\bf x}).$$

Is this a well known result, or easy to prove? Not sure how to proceed beyond the following:

$$\begin{align*} \lim_{\sigma \rightarrow 0} \nabla_{\bf x} \mathbb{E}_{\epsilon \sim \mathcal{N}(0, \sigma^2 I)}[{f({\bf x} + \epsilon)}] &= \lim_{\sigma \rightarrow 0} \int_{\epsilon} \nabla_{\bf x} f({\bf x} + \epsilon) p(\epsilon) d \epsilon \end{align*}. $$

Any tips or sources would be greatly appreciated.


1 Answer 1


I think this is true when $f$ has a continuous gradient, in which case $$\lim_{h \rightarrow \vec0} \nabla_x f(x+h) = \nabla_x f(x)$$ which is equivalent to $$\lim_{\sigma \rightarrow 0} \nabla_x f(x+\sigma z) = \nabla_x f(x)$$ for any value of $z$.

Then $$\begin{align}&\quad\ \lim_{\sigma \rightarrow 0} \nabla_x \mathbb{E}_\epsilon \left[f(x+\epsilon )\right] \\ &= \lim_{\sigma \rightarrow 0} \nabla_x \mathbb{E}_{z \sim \mathcal{N}(0,I)}\left[f(x + \sigma z)\right] \\ &= \mathbb{E}_{z \sim \mathcal{N}(0,I)}\left[\lim_{\sigma \rightarrow 0} \nabla_x f(x + \sigma z) \right] \\ &= \mathbb{E}_z \left[ \nabla_x f(x)\right] \\ & = \nabla_x f(x)\end{align}$$

  • $\begingroup$ (+1) Of course, some further technicalities on $f$ are needed to formally exchange the limit and the expectation, but this is surely the level of technicality at which the GAN paper was thinking about it. $\endgroup$
    – Danica
    Commented Feb 4, 2019 at 17:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.