In policy gradient reinforcement learning we can use a loss function of the form -log(P)*reward, where P is the probability of the selected action given the policy. For discrete actions, this turns into categorical_crossentropy:

Loss = categorical_crossentropy(chosen_action, policy_output)*reward

Some people seem to think this is just fine with negative rewards, but others not. In application, the issue I'm seeing is that hugely negative losses can be achieved. If the reward is, say, -1.0, then the loss is unboundedly negative. If the policy can be pushed to nearly zero for the chosen action, the loss can go nearly to minus infinity. For a reward of +1.0, the loss can only go down to zero. I think this asymmetry is problematic.

How should this be managed?


1 Answer 1


enter image description here

The loss functions are indeed asymmetric. But the derivatives of the loss functions are similar in both of the cases. If you have a look at the contour plot of the derivative of binary-cross entropy, you will observe that it is symmetric along the x=y line (they have the same magnitude but of opposite sign). As the gradient is similar, their effect on the learning algorithm is also similar.

  • $\begingroup$ What exactly is being plotted here? $\endgroup$
    – Mastiff
    Jul 27, 2021 at 15:13
  • $\begingroup$ The contour plot of the the gradient of binary-cross entropy with y is the target distribution and x is the predicted distribution. $\endgroup$ Jul 27, 2021 at 16:49
  • $\begingroup$ Got it. For most normal applications, the target is 0 or 1, so we could just look at that single axis (x). If the target is one, binary_ce is -log(p), where p is the network output (0 to 1). The gradient goes as -1/p. So, it's highly asymmetric and of infinite magnitude at the extreme. If the reward is -1, then the loss is log(p) and the gradient goes as 1/p. The optimizer wants to push p to zero, and the closer it gets, the bigger the gradient. I think this is all correct... $\endgroup$
    – Mastiff
    Jul 27, 2021 at 17:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.