I am training OpenAI's implementation of the A2C algorithm found here. Based on the mean episode reward graph below we can see it is in fact learning the policy function up until roughly 2000 updates:

mean episode reward

However, after roughly 1000 updates we see the overall loss explode as well as the value function loss:

Overall Loss:

overall loss

Value Function Loss:

value function loss

Also note the explosion of mean value across states:

mean value across states

Is the sudden "explosion" of overall loss a normal characteristic of A2C? Most of this significant jump seems to be attributed to the value function loss.

Note: Based on domain knowledge I know our value function is relatively hard to approximate and thus why we chose the policy gradient method A2C.

  • $\begingroup$ the loss from RL are notoriously noisy and unstable, so as long as the policy is actually improving in terms of actual reward I wouldn't worry too much $\endgroup$
    – shimao
    May 15, 2018 at 21:12
  • $\begingroup$ Thanks @shimao. I guess I'm concerned because I was expecting training to last for millions of episodes, rather I am seeing convergence after 1k updates (1 minute wall time). $\endgroup$
    – sccrthlt
    May 15, 2018 at 21:17
  • $\begingroup$ but it doesn't look like it's converged yet? $\endgroup$
    – shimao
    May 15, 2018 at 21:18
  • $\begingroup$ @shimao sorry I should have included more episodes in the graphs. The overall loss diverges after ~4.5k updates and onward. $\endgroup$
    – sccrthlt
    May 15, 2018 at 21:23

1 Answer 1


It might be connected to the entropy regularisation. If you're using A2C from baselines implementation, it has entropy*ent_coeff term for the loss.

Entropy flattens your distribution making std grow. Track the standard deviation and if it's growing, this might be an issue. Or just try to set ent_coeff to zero.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.