5
$\begingroup$

there is something I don't understand in the PyTorch implementation of Cross Entropy Loss.

As far as I understand, theoretical Cross Entropy Loss is taking log-softmax probabilities and output a real that should be closer to zero as the output is close to the target (https://ml-cheatsheet.readthedocs.io/en/latest/loss_functions.html#cross-entropy for reference)

Yet the following puzzles me:

>>> output=torch.tensor([[0.0,1.0,0.0]]) #Activation is only on the correct class
>>> target=torch.tensor([1])
>>> loss=torch.nn.CrossEntropyLoss()
>>> loss(output,target)
tensor(0.5514)

From my understanding, loss(output,target) should yield 0.0, since this is the textbook example of a 100% confident neural network. The formula given in https://pytorch.org/docs/stable/nn.html#crossentropyloss does not convince me on how it is strictly equivalent to the theoretical definition of cross entropy loss.

Is this a problem that my loss function is not equal to 0 when my model's outputs are showing 100% confidence?

$\endgroup$
  • $\begingroup$ so you should be asking on stackexchange; but the point (as doc you link mentions) is that the input expected (=output in your code) are scores which then have to be passed through softmax ie pytorch is doing logloss(softmax(scores), target), where scores are the weighted sum values. $\endgroup$ – seanv507 Mar 11 '19 at 17:27
3
$\begingroup$

The documentation says that this loss function is computed using the logloss of the softmax of $x$ (output in your code). For your example, we have $$ \begin{align} -\log\left(\frac{\exp(x_j)}{\sum_i \exp (x_i)}\right)&= -x_j+\log\left(\sum_i \exp(x_i)\right) \\ &= -1 + \log\left( \exp(0) + \exp(1) + \exp(0) \right) \\ &= 0.5514. \end{align} $$

To achieve the desired result, you could either have your network output scores as described in the documentation, or else use a loss function that works directly with probabilities.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.