1
$\begingroup$

I am having some trouble understanding how Cross Entropy would work with backpropagation. For backpropagation we exploit the chain rule to find the partial derivative of the Error function in terms of the weight. This would mean that we need the derivative of the Cross Entropy function just as we would do it with the Mean Squared Error. If I differentiate log loss I get a function which is non-defined for some values. I assume that is not acceptable as it would try to update the weight with NaN.

Cross-Entropy can be written as the following for one instance:

enter image description here

(Here x denotes the predicted value by the network, while y is the label.)

When we did differentiation with the Mean Squared Error, we simply got back

(x-y)

It is well-defined for all values. Now, if we were to differentiate the aforementioned function it would yield the following equation:

enter image description here

We can easily divide by zero if our values are actually correct in the case of binary regression. That is why I assume that this is not exactly how we deal with Cross-Entropy. Or am I wrong? Now it might be that I am getting confused but at a lot of sites this is combined with the Softmax function.

Is the process a little different than it was with MSE? Meaning that we first find the derivative of the Cross-Entropy function just as we would do with MSE and then everything is the same?

enter image description here

$\endgroup$
  • $\begingroup$ Can you edit your post to elaborate on what you mean by "undefined for some values"? What expression are you using for cross entropy and its derivative? Which values are you concerned about? $\endgroup$ – Sycorax Sep 22 '19 at 14:44
  • $\begingroup$ @Sycorax I tried to edit it so it is clearer, I hope it helps. $\endgroup$ – Szász Erik Sep 22 '19 at 15:16
1
$\begingroup$

For any finite input, softmax outputs are strictly between 0 and 1. This implies that you'll never divide by zero. This is one reason that you've observed softmax and cross-entropy are commonly used together.

Finite-precision arithmetic can result in numerical underflow, especially when round-tripping exponentiation. This can be avoided by working on the logit scale directly; for example: http://pytorch.org/docs/stable/nn.html#crossentropyloss

See also: Infinities with cross entropy in practice

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ How exactly is that? I included the derivative of the error function, Cross-Entropy. As I can get values between 0 and 1 it is more than possible that (x-1)x will be 0 as after a while the computer will approximate very small numbers that tend to 0,well to 0. Then I would get an NaN which would ruin the whole calculation. $\endgroup$ – Szász Erik Sep 22 '19 at 15:39
  • $\begingroup$ Also related to the other part: So when backpropagating I just change the derivative of the MSE function to the derivative of the Cross-Entropy and that's it? $\endgroup$ – Szász Erik Sep 22 '19 at 15:40
  • 1
    $\begingroup$ You're correct that finite-precision arithmetic has limitations, but that wasn't asked in a your question. Anyway, if you're concerned about numerical stability, then it's common to work with logits directly, such as pytorch.org/docs/stable/nn.html#crossentropyloss which avoids the precision issues that can arise when round-tripping $\exp$ and $\log$. Yes, backprop only depends on the functions that you use. $\endgroup$ – Sycorax Sep 22 '19 at 15:44
  • $\begingroup$ A different workaround is to make sure that the output of the softmax is strictly between $(\epsilon, 1-\epsilon)$ with $\epsilon=0.0001$ or so. $\endgroup$ – bayerj Sep 22 '19 at 18:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.