I am having some trouble understanding how Cross Entropy would work with backpropagation. For backpropagation we exploit the chain rule to find the partial derivative of the Error function in terms of the weight. This would mean that we need the derivative of the Cross Entropy function just as we would do it with the Mean Squared Error. If I differentiate log loss I get a function which is non-defined for some values. I assume that is not acceptable as it would try to update the weight with NaN.

Cross-Entropy can be written as the following for one instance:

enter image description here

(Here x denotes the predicted value by the network, while y is the label.)

When we did differentiation with the Mean Squared Error, we simply got back


It is well-defined for all values. Now, if we were to differentiate the aforementioned function it would yield the following equation:

enter image description here

We can easily divide by zero if our values are actually correct in the case of binary regression. That is why I assume that this is not exactly how we deal with Cross-Entropy. Or am I wrong? Now it might be that I am getting confused but at a lot of sites this is combined with the Softmax function.

Is the process a little different than it was with MSE? Meaning that we first find the derivative of the Cross-Entropy function just as we would do with MSE and then everything is the same?

enter image description here

  • $\begingroup$ Can you edit your post to elaborate on what you mean by "undefined for some values"? What expression are you using for cross entropy and its derivative? Which values are you concerned about? $\endgroup$
    – Sycorax
    Sep 22, 2019 at 14:44
  • $\begingroup$ @Sycorax I tried to edit it so it is clearer, I hope it helps. $\endgroup$ Sep 22, 2019 at 15:16

1 Answer 1


For any finite input, softmax outputs are strictly between 0 and 1. This implies that you'll never divide by zero. This is one reason that you've observed softmax and cross-entropy are commonly used together.

Finite-precision arithmetic can result in numerical underflow, especially when round-tripping exponentiation. This can be avoided by working on the logit scale directly; for example: http://pytorch.org/docs/stable/nn.html#crossentropyloss

See also: Infinities with cross entropy in practice

  • $\begingroup$ How exactly is that? I included the derivative of the error function, Cross-Entropy. As I can get values between 0 and 1 it is more than possible that (x-1)x will be 0 as after a while the computer will approximate very small numbers that tend to 0,well to 0. Then I would get an NaN which would ruin the whole calculation. $\endgroup$ Sep 22, 2019 at 15:39
  • $\begingroup$ Also related to the other part: So when backpropagating I just change the derivative of the MSE function to the derivative of the Cross-Entropy and that's it? $\endgroup$ Sep 22, 2019 at 15:40
  • 1
    $\begingroup$ You're correct that finite-precision arithmetic has limitations, but that wasn't asked in a your question. Anyway, if you're concerned about numerical stability, then it's common to work with logits directly, such as pytorch.org/docs/stable/nn.html#crossentropyloss which avoids the precision issues that can arise when round-tripping $\exp$ and $\log$. Yes, backprop only depends on the functions that you use. $\endgroup$
    – Sycorax
    Sep 22, 2019 at 15:44
  • $\begingroup$ A different workaround is to make sure that the output of the softmax is strictly between $(\epsilon, 1-\epsilon)$ with $\epsilon=0.0001$ or so. $\endgroup$
    – bayerj
    Sep 22, 2019 at 18:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.