# Is there a typo in the paper "Evolution Strategies as a Scalable Alternative to reinforcement learning" by OpenAI?

Original paper: https://arxiv.org/pdf/1703.03864.pdf

On page 2-3, it writes,

But this equation is clearly wrong.

Since the step needed to derive the equation involves the argument, let $$\theta = \psi + \sigma \epsilon, \epsilon \sim N(0,I)$$

Therefore by a straight forward substitution, $$\mathbb{E}_{\theta \sim p_\psi} F(\theta) = \mathbb{E}_{\epsilon \sim N(0,I)} F(\psi + \sigma\epsilon)$$

and NOT,

$$\mathbb{E}_{\theta \sim p_\psi} F(\theta) =\mathbb{E}_{\epsilon \sim N(0,I)} F(\theta + \sigma\epsilon)$$ ($$\theta$$ and $$\psi$$ exchanged) as written in the paper. Hence all the rest of the results are wrong.

One can argue that the authors, confusingly, implicitly exchanged the notation of $$\psi$$ and $$\theta$$, where now $$\theta$$ refers to the mean of the Gaussian distribution (which was previously denoted by $$\psi$$)

However, this implies that their main algorithm is wrong,

Because $$\theta$$ here is clearly not the mean of the Gaussian. $$\theta$$ clearly refers to the neural network parameter (weights) in the original paper.

Can someone check for me?

Note that this error is also pointed out in an earlier post How is the equation in "Evolution Strategies as a Scalable Alternative to Reinforcement Learning" derived? however, in my opinion this is not just an abuse of notation "confusing" but it is flat out wrong.

• Does this answer your question? How is the equation in "Evolution Strategies as a Scalable Alternative to Reinforcement Learning" derived? Commented Jan 4, 2020 at 8:20
• @Xi'an No. The answer said "$\theta$ to be the "mean parameter vector". But as I have pointed out in my question, the way that $\theta$ is updated in algorithm 1 does not imply that $\theta$ is the mean parameter vector, but the original weights of the neural network. Unless, we take the weights of the neural network to be the mean of a Gaussian....which the paper does not claim. Commented Jan 4, 2020 at 20:35

• For the first question, I think the authors just exchange the notations $$\psi$$ and $$\theta$$.
• For the second question, the weights of neural networks are modeled as Gaussian variables with distribution $$\mathcal{N}(\theta, \sigma^2I)$$.