# Reinforcement learning: is a softmax policy actor-critic expected to work on mountain car?

I am following David Silver's RL course and I'm struggling to apply the Actor Critic concept to the Mountain Car environment. I am using a softmax policy with linear function approximation. I am also estimating the action value function with linear function approximation. In math:

• $$\phi(s, a)\sim featurizer([s_1 \space s_2], [a])$$
• $$Q(s, a) = w· \phi(s, a)$$
• $$\pi(a|s, \theta) = \frac{e^{\theta \phi(s_, a)}}{\sum_{a'\in{A}} {e^{\theta\phi(s_, a')}}}$$
• $$\Delta w_{each\space loop} \sim TD(0)$$
• $$\Delta\theta_{each\space loop} = \alpha_\theta · (e^{\theta\phi(s_, a)}- \frac{\sum_{a'\in{A}} {e^{\theta\phi(s_, a')}}\phi(s_, a)}{\sum_{a'\in{A}} {e^{\theta\phi(s_, a')}}})·Q(s, a)$$

Concisely, my problem so far lies in the fact that my algorithm breaks, apparently due to the unstability of $$\theta$$. It remains generally stable, but sometimes exceptional updates happen where it sees some of its values jump a few orders of magnitude. This consistently breaks my algorithm somewhere between the 1st-20th episode.

I have looked in the internet for similar cases but everyone seems to be using neural nets and whatnot, while I am attempting the most basic thing I could think of. I have also tried some rough fixes on the code (with no success) but I feel that this method should be able to work without 'hacks'.

I do not understand why these diverging policy updates happen, and whether this might be some dumb mistake or it just won't work because I am taking an unsuitable approach to Mountain Car. Can someone shed some light on this for me, please?

Find my code here.