# Cross Entropy in PyTorch is different from what I learnt (Not about logit input, but about the loss for every node)

I know that the CrossEntropyLoss in Pytorch expects logits. I also know that the reduction argument in CrossEntropyLoss is to reduce along the data sample's axis, if it is reduction=mean, that is to take $$\frac{1}{m}\sum^m_{i=1}$$. If reduction=sum, then it is $$\sum^m_{i=1}$$. If I use 'none', it will just give me a tensor list of loss of each data sample fed. So I am not asking about these two things.

My problem is that about the fact that what I have learnt about cross entropy is to calculate the loss for every output node. But in PyTorch, it only calculates for the class fed.

PyTorch's CrossEntropyLoss has a reduction argument, but it is to do mean or sum or none over the data samples axis.

Assume I am doing everything from scratch, that now I have a model, with 3 output nodes (data has 3 classes $$C=3$$), and I only pass one data sample $$m=1$$ to the model. I call the logits of the three output nodes $$z_1,z_2,z_3$$. If I code everything from scratch I am going to do softmax on them and I will obtain $$\hat{y}_1,\hat{y}_2,\hat{y}_3$$. Assume the label of this data sample is 0, so I convert it into one-hot label as $$[y_1,y_2,y_3]=[1,0,0]$$. After that I can use cross entropy to compute

$$\sum^C_{k=1}[-y_k\log \hat{y}_k-(1-y_k)\log (1-\hat{y}_k)]$$

So, for the final loss for gradient descent, i will sum all the 3 cross entropy loss for each node.

But in PyTorch, it will only calculate the one with the class 0 as the label for this data sample is 0

$$-y_1\log \hat{y}_1-(1-y_1)\log (1-\hat{y}_1)$$

and ignore others

Why is that?

To show it in code

• Please don't delete and re-post closed questions. Doing so destroys all of the work that others put into clarifying comments. Instead, edit the closed question to clarify what it is that you'd like to know more about. In this case, the answer to your question is contained in my comment to your previous post: the loss for a single sample is a sum, so it's entirely consistent to return a single loss value for a single instance.
– Sycorax
Jul 14, 2021 at 16:14
• No now I only have one data sample. I mean I have 3 output nodes. So the cross-entropy should calculate the loss for all three nodes then sum them up. But it only calculates for the 0th node's loss because the label is 0. Jul 14, 2021 at 16:20
• i mean, z is not three data samples, it is one data sample. z is the output node's logits Jul 14, 2021 at 16:21
• Re-read what I wrote. The loss of a single sample is the sum of the three nodes' predictions. But the expression $\sum_j y_j \log p_j$ has only one nonzero term, because only one element of $y$ is nonzero.
– Sycorax
Jul 14, 2021 at 16:21
• For a three label problem, the expression you've written isn't the correct loss, so perhaps that's the misunderstanding.
– Sycorax
Jul 14, 2021 at 16:26

The loss of a single sample is the sum of the $$J$$ nodes' predictions, so for each sample, the loss is a scalar value.
The expression $$-\sum_j^J y_j \log p_j$$ has only one nonzero term, because only one element of $$y$$ is nonzero. This is the correct loss for a classification problem; we can prove that this is correct by writing out the likelihood of the categorical distribution and taking the negative of its logarithm to get the cross-entropy. It's merely a special case that we can re-write the binary classification problem in terms of one probability $$p$$ and one binary label $$y$$ in the convenient form $$-y \log p - (1-y)\log (1-p)$$.
Let's write out a concrete example. We have predictions $$[0.4,0.3,0.2]$$ and label $$[1,0,0]$$. By inspection, we know that the loss is $$-\log(0.4)$$ because this is the only entry with a nonzero label.