# Pytorch Cross Entropy Loss implementation counterintuitive

there is something I don't understand in the PyTorch implementation of Cross Entropy Loss.

As far as I understand, theoretical Cross Entropy Loss is taking log-softmax probabilities and output a real that should be closer to zero as the output is close to the target (https://ml-cheatsheet.readthedocs.io/en/latest/loss_functions.html#cross-entropy for reference)

Yet the following puzzles me:

>>> output=torch.tensor([[0.0,1.0,0.0]]) #Activation is only on the correct class
>>> target=torch.tensor([1])
>>> loss=torch.nn.CrossEntropyLoss()
>>> loss(output,target)
tensor(0.5514)



From my understanding, loss(output,target) should yield 0.0, since this is the textbook example of a 100% confident neural network. The formula given in https://pytorch.org/docs/stable/nn.html#crossentropyloss does not convince me on how it is strictly equivalent to the theoretical definition of cross entropy loss.

Is this a problem that my loss function is not equal to 0 when my model's outputs are showing 100% confidence?

• so you should be asking on stackexchange; but the point (as doc you link mentions) is that the input expected (=output in your code) are scores which then have to be passed through softmax ie pytorch is doing logloss(softmax(scores), target), where scores are the weighted sum values. Commented Mar 11, 2019 at 17:27

The documentation says that this loss function is computed using the logloss of the softmax of $$x$$ (output in your code). For your example, we have \begin{align} -\log\left(\frac{\exp(x_j)}{\sum_i \exp (x_i)}\right)&= -x_j+\log\left(\sum_i \exp(x_i)\right) \\ &= -1 + \log\left( \exp(0) + \exp(1) + \exp(0) \right) \\ &= 0.5514. \end{align}