# Why is random sampling a non-differentiable operation?

This answer states that we cannot back-propagate through a random node. So, in the case of VAEs, you have the reparametrisation trick, which shifts the source of randomness to another variable different than $$z$$ (the latent vector), so that you can now differentiate with respect to $$z$$. Similarly, this question states that we cannot differentiate a random sampling operation.

Why exactly is this the case? Why is randomness a problem when differentiating and back-propagating? I think this should be made explicit and clear.

What does a “random node” mean and what does it mean for backprop to “flow” or not flow through such a node?

## Undifferentiable expectations

Let’s say we want to take the gradient w.r.t. $$\theta$$ of the following expectation, $$\mathbb{E}_{p(z)}[f_{\theta}(z)]$$ where $$p$$ is a density. Provided we can differentiate $$f_{\theta}(x)$$, we can easily compute the gradient:

\begin{align} \nabla_{\theta} \mathbb{E}_{p(z)}[f_{\theta}(z)] &= \nabla_{\theta} \Big[ \int_{z} p(z) f_{\theta}(z) dz \Big] \\ &= \int_{z} p(z) \Big[\nabla_{\theta} f_{\theta}(z) \Big] dz \\ &= \mathbb{E}_{p(z)} \Big[\nabla_{\theta} f_{\theta}(z) \Big] \end{align}

In words, the gradient of the expectation is equal to the expectation of the gradient. But what happens if our density $$p$$ is also parameterized by $$\theta$$?

\begin{align} \nabla_{\theta} \mathbb{E}_{p_{\theta}(z)}[f_{\theta}(z)] &= \nabla_{\theta} \Big[ \int_{z} p_{\theta}(z) f_{\theta}(z) dz \Big] \\ &= \int_{z} \nabla_{\theta} \Big[ p_{\theta}(z) f_{\theta}(z) \Big] dz \\ &= \int_{z} f_{\theta}(z) \nabla_{\theta} p_{\theta}(z)dz + \int_{z} p_{\theta}(z) \nabla_{\theta} f_{\theta}(z)dz \\ &= \underbrace{\int_{z} f_{\theta}(z) \nabla_{\theta} p_{\theta}(z)}_{\text{What about this?}}dz + \mathbb{E}_{p_{\theta}(z)} \Big[f_{\theta}(z)\Big] \end{align}

The first term of the last equation is not guaranteed to be an expectation. Monte Carlo methods require that we can sample from $$p_{\theta}(z)$$, but not that we can take its gradient. This is not a problem if we have an analytic solution to $$\nabla_{\theta}p_{\theta}(z)$$, but this is not true in general. 1

• The last, the penultimate and the one before the penultimate equalities are not clear at all. 1. Why can we bring the gradient into the integral? 2. Why does the integral distribute like a product rule for differentiation in this case? 3. What is the relation between the fact that the first term of the last formula is not guaranteed to be an expectation and the fact that we cannot propagate through a random node? 4. I don't get how the second term of the last formula becomes an expectation over $f$ and not over $\nabla f$. – nbro May 24 at 23:19
• @nbro: Did you finally understood this? I'm stuck exactly at this and cant figure out! – Rika Oct 3 at 2:28
• @Breeze The answer to question 1 in my comment above is: the gradient is being taken with respect to $\theta$, while the integral is over the variable $z$, so $\nabla_\theta$ acts like a constant with respect to the integral, so it can be brought inside the integral (the "constant rule for integration"). The answer to question 2 is: We apply the gradient to a product of two functions, so we apply the product rule for differentiation. Then we will have an integral over a sum of functions. Given that the integral can be distributed over sums, then we obtain the penultimate term. – nbro Oct 3 at 2:48
• @Breeze In this answer, it seems like we are showing that the gradient of an expectation with respect to density or random variable is somehow not well-defined, but I am still not sure myself. So I still don't know the exact answer to my 3rd and 4th questions. – nbro Oct 3 at 2:53
• Thanks a lot, I really appreciate your kind explanation – Rika Oct 3 at 5:08