I recently came across the following curious identity:

$$\nabla_\theta \mathbb{E}_{x \sim D_\theta}[f(x)] = \mathbb{E}_{x \sim D_\theta} [ \nabla_\theta \log(D_\theta(x)) f(x)],$$

where $D_\theta$ represents a probability distribution parametrized by $\theta$, $D_\theta(x)$ represents the probability that this distribution assigns to outcome $x$, $\nabla_\theta$ represents the gradient with respect to $\theta$, and $f$ represents some arbitrary function.

I can prove algebraically why this identity holds. However, I lack intuition. Is there any intuition for why this should hold? Perhaps something to understand why it is natural for the logarithm of the probability to appear inside the expected value, or an interpretation of these quantities that makes it natural why the identity would hold?

Here's the algebraic derivation:

$$\nabla_\theta \log(D_\theta(x)) f(x) = {\nabla_\theta D_\theta(x) \over D_\theta(x)} f(x),$$


$$\begin{align*} \mathbb{E}_{x \sim D_\theta} [ \nabla_\theta \log(D_\theta(x)) f(x)] &= \sum_x D_\theta(x) \nabla_\theta \log(D_\theta(x)) f(x)\\ &= \sum_x D_\theta(x) {\nabla_\theta D_\theta(x) \over D_\theta(x)} f(x)\\ &= \sum_x \nabla_\theta D_\theta(x) f(x)\\ &= \nabla_\theta \sum_x D_\theta(x) f(x)\\ &= \nabla_\theta \mathbb{E}_{x \sim D_\theta}[f(x)]. \end{align*}$$


It is the outcome of two tricks often used in analysis

  1. the log derivative identity (with $p(x; \theta)$ denoting the density of $D_\theta$) $$ \nabla_\theta \log p (x; \theta) = \frac{\nabla_\theta p (x; \theta)}{p(x; \theta)} $$
  2. exchange the orders of derivative and integral

The logarithm appears just because one wants to write the fraction of gradient $p$ by $p$ in a nice form.


This reminds me of something called Fisher's identity, which states $$ \nabla_{\theta} \log p(y \mid \theta) = \int \nabla_{\theta}\log p(x,y \mid \theta)p(x\mid y,\theta) \text{d} x. $$ This is useful for estimating latent variable models because it writes the derivative of the log likelihood (a quantity that is often intractable owing to the fact it is the derivative of a high-dimensional integral) in terms of an expectation with respect to the smoothing distribution. The integrand of the latter is is the log of the complete data likelihood, which is a bunch of things that are added together, making it is easy to exploit linearity.

If you set $f(x) = 1$ and $D_{\theta}(x) = p(x \mid y, \theta)$, then you can write your equation to look more like mine. However, I suppose your equation is a bit more general because $f$ might be a little more interesting. Do you mind sharing where this popped up?

  • 1
    $\begingroup$ Interesting, thank you. One difference is that it looks like Fisher's identity has a log on both sides, whereas in the identity I'm asking about, there is no log on the left-hand side. I came across this while reading about reinforcement learning; see e.g. equation (1) at gradientscience.org/policy_gradients_pt1. $\endgroup$ – D.W. Feb 11 '19 at 17:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.