Why is OpenAI's evolution strategy a natural evolution strategy?

Natural Evolution Strategies follow the natural gradient using the Fisher Information Matrix $$\mathbf{F}_\theta$$ of a search distribtion $$p_\theta$$. That is, parameters in natural evolution strategies are updated with:

$$$$\theta \gets \theta + \alpha \mathbf{F}^{-1} \nabla_\theta J$$$$

with

\begin{align} \nabla_\theta J =& \frac{1}{n} \sum_{i=1}^n f(x_i) \nabla_\theta p_\theta(x_i)\\ \mathbf{F} = & \frac{1}{n}\nabla_\theta\log p_\theta(x_i) \nabla_\theta\log p_\theta(x_i)^\top \end{align}

where $$x_i$$ is a drawn sample from $$p_\theta(.)$$ and $$f(x_i)$$ is its fitness.

In OpenAI's evolution strategy approach the update looks like this:

$$$$\theta \gets \theta + \alpha \nabla_\theta J$$$$

with

$$$$\nabla_\theta J = \frac{1}{n \sigma} \sum_{i=1}^n f(\theta + \sigma \epsilon_i) \epsilon_i$$$$

I do not quite understand, why the latter is a natural evolution strategy, i.e., follow the natural gradient, as the authors claim.

• Did you end up coming up with an explanation to your question? Aug 27, 2022 at 13:13
• It boils down to usage of ${\it natural gradient}$, Amari's definition used KL distance and the first paper you mentioned used Fisher's matrix. It is about accounting for uncertainty in second order updates.In OpenAI's approach using $\sigma$, noise standard-deviation accounts for the uncertainty, so it makes the apprach natural evaluation strategy. Aug 28, 2022 at 13:43
• @msuzen I don't think that's correct as stated. Yes, the natural gradient may be about accounting for uncertainty, but it is about accounting for it in a very particular way. Not any constant factor depending on the standard-deviation makes the method a NES. In contrast to the update rule used by OpenAI, using the Fisher information would give you a sigma in the numerator and not in the denominator. Aug 29, 2022 at 10:58
• @s1624210 Yes, you are right. It looks like there might be some intricacies and deeper insights there. Though, the natural gradient is indeed about accounting for uncertainty. Aug 29, 2022 at 16:51