I know that the CrossEntropyLoss
in Pytorch expects logits. I also know that the reduction argument in CrossEntropyLoss
is to reduce along the data sample's axis, if it is reduction=mean
, that is to take $\frac{1}{m}\sum^m_{i=1}$. If reduction=sum
, then it is $\sum^m_{i=1}$. If I use 'none', it will just give me a tensor list of loss of each data sample fed. So I am not asking about these two things.
My problem is that about the fact that what I have learnt about cross entropy is to calculate the loss for every output node. But in PyTorch, it only calculates for the class fed.
PyTorch's CrossEntropyLoss
has a reduction argument, but it is to do mean or sum or none over the data samples axis.
Assume I am doing everything from scratch, that now I have a model, with 3 output nodes (data has 3 classes $C=3$), and I only pass one data sample $m=1$ to the model. I call the logits of the three output nodes $z_1,z_2,z_3$. If I code everything from scratch I am going to do softmax on them and I will obtain $\hat{y}_1,\hat{y}_2,\hat{y}_3$. Assume the label of this data sample is 0, so I convert it into one-hot label as $[y_1,y_2,y_3]=[1,0,0]$. After that I can use cross entropy to compute
$\sum^C_{k=1}[-y_k\log \hat{y}_k-(1-y_k)\log (1-\hat{y}_k)]$
So, for the final loss for gradient descent, i will sum all the 3 cross entropy loss for each node.
But in PyTorch, it will only calculate the one with the class 0 as the label for this data sample is 0
$-y_1\log \hat{y}_1-(1-y_1)\log (1-\hat{y}_1)$
and ignore others
Why is that?