# How does Cross-Entropy (log loss) work with backpropagation?

I am having some trouble understanding how Cross Entropy would work with backpropagation. For backpropagation we exploit the chain rule to find the partial derivative of the Error function in terms of the weight. This would mean that we need the derivative of the Cross Entropy function just as we would do it with the Mean Squared Error. If I differentiate log loss I get a function which is non-defined for some values. I assume that is not acceptable as it would try to update the weight with NaN.

Cross-Entropy can be written as the following for one instance:

(Here x denotes the predicted value by the network, while y is the label.)

When we did differentiation with the Mean Squared Error, we simply got back

(x-y)


It is well-defined for all values. Now, if we were to differentiate the aforementioned function it would yield the following equation:

We can easily divide by zero if our values are actually correct in the case of binary regression. That is why I assume that this is not exactly how we deal with Cross-Entropy. Or am I wrong? Now it might be that I am getting confused but at a lot of sites this is combined with the Softmax function.

Is the process a little different than it was with MSE? Meaning that we first find the derivative of the Cross-Entropy function just as we would do with MSE and then everything is the same?

• Can you edit your post to elaborate on what you mean by "undefined for some values"? What expression are you using for cross entropy and its derivative? Which values are you concerned about?
– Sycorax
Sep 22 '19 at 14:44
• @Sycorax I tried to edit it so it is clearer, I hope it helps. Sep 22 '19 at 15:16

• You're correct that finite-precision arithmetic has limitations, but that wasn't asked in a your question. Anyway, if you're concerned about numerical stability, then it's common to work with logits directly, such as pytorch.org/docs/stable/nn.html#crossentropyloss which avoids the precision issues that can arise when round-tripping $\exp$ and $\log$. Yes, backprop only depends on the functions that you use.
• A different workaround is to make sure that the output of the softmax is strictly between $(\epsilon, 1-\epsilon)$ with $\epsilon=0.0001$ or so. Sep 22 '19 at 18:27