1
$\begingroup$

I am following David Silver's RL course and I'm struggling to apply the Actor Critic concept to the Mountain Car environment. I am using a softmax policy with linear function approximation. I am also estimating the action value function with linear function approximation. In math:

  • $\phi(s, a)\sim featurizer([s_1 \space s_2], [a])$
  • $Q(s, a) = w· \phi(s, a)$
  • $\pi(a|s, \theta) = \frac{e^{\theta \phi(s_, a)}}{\sum_{a'\in{A}} {e^{\theta\phi(s_, a')}}}$
  • $\Delta w_{each\space loop} \sim TD(0) $
  • $\Delta\theta_{each\space loop} = \alpha_\theta · (e^{\theta\phi(s_, a)}- \frac{\sum_{a'\in{A}} {e^{\theta\phi(s_, a')}}\phi(s_, a)}{\sum_{a'\in{A}} {e^{\theta\phi(s_, a')}}})·Q(s, a)$

Concisely, my problem so far lies in the fact that my algorithm breaks, apparently due to the unstability of $\theta$. It remains generally stable, but sometimes exceptional updates happen where it sees some of its values jump a few orders of magnitude. This consistently breaks my algorithm somewhere between the 1st-20th episode.

I have looked in the internet for similar cases but everyone seems to be using neural nets and whatnot, while I am attempting the most basic thing I could think of. I have also tried some rough fixes on the code (with no success) but I feel that this method should be able to work without 'hacks'.

I do not understand why these diverging policy updates happen, and whether this might be some dumb mistake or it just won't work because I am taking an unsuitable approach to Mountain Car. Can someone shed some light on this for me, please?

Find my code here.

Thanks in advance!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.