How is the equation in "Evolution Strategies as a Scalable Alternative to Reinforcement Learning" derived? In the OpenAI paper "Evolution Strategies as a Scalable Alternative to Reinforcement Learning", how is the equation in page 3 derived? Thanks. 

 A: The score function estimator is
$$\nabla_\psi \mathbb{E}_{\theta \sim p_\psi} F(\theta) = \mathbb{E}_{\theta \sim p_\psi} \left[ F(\theta) \nabla_\psi \log p_\psi(\theta)\right]$$
We have $p_\psi$ as a gaussian with mean $\psi$ and covariance $\sigma^2 I$. Therefore:
$$\nabla_\psi \log p_\psi(\theta) = \nabla_\psi \frac{(\theta-\psi)^2}{\sigma^2} = \frac{\theta-\psi}{\sigma^2}$$
(this leaves out a bunch of constant factors for simplicity). Recall that $\theta \sim p_\psi$ so we can rewrite it as $\theta = \psi + \sigma \epsilon$, where $\epsilon \sim \mathcal{N}(0, I)$. Therefore:
$$\frac{\theta-\psi}{\sigma^2} = \frac{\epsilon}{\sigma}$$
Putting this back into the RHS of the score function estimator gives us:
$$\nabla_\psi \mathbb{E}_{\theta \sim p_\psi} F(\theta) = 
\frac{1}{\sigma} \mathbb{E}_{\theta \sim p_\psi} \left[ F(\theta) \epsilon \right] = \frac{1}{\sigma} \mathbb{E}_{\epsilon \sim \mathcal{N}(0, I)} \left[ F(\psi+\sigma \epsilon) \epsilon \right]$$
The authors of the paper decided to change the meaning of $\theta$ to be the "mean parameter vector", replacing $\psi$, which in my opinion is fairly confusing. Other than that detail, we have arrived at the same expression.
