# A2C Loss Function Explosion

I am training OpenAI's implementation of the A2C algorithm found here. Based on the mean episode reward graph below we can see it is in fact learning the policy function up until roughly 2000 updates:

However, after roughly 1000 updates we see the overall loss explode as well as the value function loss:

Overall Loss:

Value Function Loss:

Also note the explosion of mean value across states:

Is the sudden "explosion" of overall loss a normal characteristic of A2C? Most of this significant jump seems to be attributed to the value function loss.

Note: Based on domain knowledge I know our value function is relatively hard to approximate and thus why we chose the policy gradient method A2C.

• the loss from RL are notoriously noisy and unstable, so as long as the policy is actually improving in terms of actual reward I wouldn't worry too much May 15, 2018 at 21:12
• Thanks @shimao. I guess I'm concerned because I was expecting training to last for millions of episodes, rather I am seeing convergence after 1k updates (1 minute wall time). May 15, 2018 at 21:17
• but it doesn't look like it's converged yet? May 15, 2018 at 21:18
• @shimao sorry I should have included more episodes in the graphs. The overall loss diverges after ~4.5k updates and onward. May 15, 2018 at 21:23

It might be connected to the entropy regularisation. If you're using A2C from baselines implementation, it has entropy*ent_coeff term for the loss.
Entropy flattens your distribution making std grow. Track the standard deviation and if it's growing, this might be an issue. Or just try to set ent_coeff to zero.