I am following David Silver's RL course and I'm struggling to apply the Actor Critic concept to the Mountain Car environment. I am using a softmax policy with linear function approximation. I am also estimating the action value function with linear function approximation. In math:

  • $\phi(s, a)\sim featurizer([s_1 \space s_2], [a])$
  • $Q(s, a) = w· \phi(s, a)$
  • $\pi(a|s, \theta) = \frac{e^{\theta \phi(s_, a)}}{\sum_{a'\in{A}} {e^{\theta\phi(s_, a')}}}$
  • $\Delta w_{each\space loop} \sim TD(0) $
  • $\Delta\theta_{each\space loop} = \alpha_\theta · (e^{\theta\phi(s_, a)}- \frac{\sum_{a'\in{A}} {e^{\theta\phi(s_, a')}}\phi(s_, a)}{\sum_{a'\in{A}} {e^{\theta\phi(s_, a')}}})·Q(s, a)$

Concisely, my problem so far lies in the fact that my algorithm breaks, apparently due to the unstability of $\theta$. It remains generally stable, but sometimes exceptional updates happen where it sees some of its values jump a few orders of magnitude. This consistently breaks my algorithm somewhere between the 1st-20th episode.

I have looked in the internet for similar cases but everyone seems to be using neural nets and whatnot, while I am attempting the most basic thing I could think of. I have also tried some rough fixes on the code (with no success) but I feel that this method should be able to work without 'hacks'.

I do not understand why these diverging policy updates happen, and whether this might be some dumb mistake or it just won't work because I am taking an unsuitable approach to Mountain Car. Can someone shed some light on this for me, please?

Find my code here.

Thanks in advance!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.