I was wondering, if there was a way to implement natural actor critic without having to deal with choosing the right features for the action-state-space.

Basically, in Sutton,Barto - Reinforcement Learning, they propose an Actor-Critic algorithm, which, similarly to REINFORCE, uses two arbitrary function approximations, one for the policy and one for the value function. Then at each step the TD error gets computed as usual:

$$\delta = R + \gamma \hat{v}(S',w)-\hat v(S,w) $$ and then perform a gradient descent step for $\hat v$, as well as for $\pi(\cdot|\cdot, \theta)$, weighted by the TD error $\delta$.

This works fine for any parametrized approximators of $v$ and $\pi$. However, I was wondering if a similar approach would be feasible when using natural policy gradient updates. The idea of natural policy gradients is to form updates based on the natural gradient

$$\hat \nabla J(\pi) = G(\theta)^{-1}\nabla J(\pi)$$ where $J(\pi)$ is some performance measure of the policy and $G(\theta)$ denotes the fisher Information matrix of the policy ($\mathbb{E}[\nabla \log\pi(a|s)\nabla \log\pi(a|s)^T]$). In various papers (e.g. see https://papers.nips.cc/paper/3258-incremental-natural-actor-critic-algorithms.pdf) algorithms have been proposed which make use of the natural gradient, but all of them share that the complexity grows quadratically with the amount of parameters $\theta$ needed to parametrize the policy. This makes using neural network policies unfeasible for even very small networks.

Is there any workaround for this, or does the natural gradient approach only work for low dimensional problems, i.e. where a policy parametrization can be chosen in a way that only few parameters are needed?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.