0
$\begingroup$

I would like to implement the natural gradient for reinforcement learning as described in the following paper: https://arxiv.org/pdf/1703.02660.pdf

However, I do not know how to compute the empirical Fisher Information matrix to implement gradient ascent with the following parameter update $\theta_{t+1} := \theta_t + F^{-1}\nabla_\theta J(\pi_\theta)$, where $\nabla_\theta J(\pi_\theta)$ is the regular policy gradient weighted by the advantages.

When computing the empirical Fisher Information as the outer product of the log policy gradient $F = \frac{1}{T}\sum_{t=1}^T\nabla_{\theta}\, log\, \pi_\theta\, \nabla_{\theta}\, log\, \pi_\theta^\intercal$ (summed over all trajectories/samples), the resulting Fisher matrix is not positive semidefinite.

I do not see any reason why the policy gradient $\nabla_{\theta}\, log\, \pi\,$ should not have negative components.

What is the right way to practically calculate the empirical Fisher information from the gradient in an implementation? Is it correct to directly use the outer product of the gradients (e.g. with numpy as F = np.outer(grad, grad)?

$\endgroup$
  • 1
    $\begingroup$ I believe most, if not all, of the questions you pose may have been answered (albeit in a slightly different context) at stats.stackexchange.com/questions/7308/…. Does that help? $\endgroup$ – whuber Mar 4 '19 at 18:52
  • $\begingroup$ Thank you! This does answer some of my questions. But I am wondering whether there are better ways to implement the natural gradient than literally computing the outer product of the log policy gradient (equation 3.11 in this paper: citeseerx.ist.psu.edu/viewdoc/…). $\endgroup$ – cookiedealer Mar 5 '19 at 18:03
  • $\begingroup$ Would you mind editing your post to focus on that issue? That might encourage people to post appropriate answers. $\endgroup$ – whuber Mar 5 '19 at 18:07
  • $\begingroup$ Thanks, I updated the question, let me know if more details are necessary at any point $\endgroup$ – cookiedealer Mar 5 '19 at 18:27
0
$\begingroup$

I found an answer to my question here: http://www.telesens.co/2018/06/09/efficiently-computing-the-fisher-vector-product-in-trpo/

The second derivative of the KL divergence can be used, but there are better ways to approximate the Natural Gradient like TRPO.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.