2
$\begingroup$

I know that the CrossEntropyLoss in Pytorch expects logits. I also know that the reduction argument in CrossEntropyLoss is to reduce along the data sample's axis, if it is reduction=mean, that is to take $\frac{1}{m}\sum^m_{i=1}$. If reduction=sum, then it is $\sum^m_{i=1}$. If I use 'none', it will just give me a tensor list of loss of each data sample fed. So I am not asking about these two things.

My problem is that about the fact that what I have learnt about cross entropy is to calculate the loss for every output node. But in PyTorch, it only calculates for the class fed.

PyTorch's CrossEntropyLoss has a reduction argument, but it is to do mean or sum or none over the data samples axis.

Assume I am doing everything from scratch, that now I have a model, with 3 output nodes (data has 3 classes $C=3$), and I only pass one data sample $m=1$ to the model. I call the logits of the three output nodes $z_1,z_2,z_3$. If I code everything from scratch I am going to do softmax on them and I will obtain $\hat{y}_1,\hat{y}_2,\hat{y}_3$. Assume the label of this data sample is 0, so I convert it into one-hot label as $[y_1,y_2,y_3]=[1,0,0]$. After that I can use cross entropy to compute

$\sum^C_{k=1}[-y_k\log \hat{y}_k-(1-y_k)\log (1-\hat{y}_k)]$

So, for the final loss for gradient descent, i will sum all the 3 cross entropy loss for each node.

But in PyTorch, it will only calculate the one with the class 0 as the label for this data sample is 0

$-y_1\log \hat{y}_1-(1-y_1)\log (1-\hat{y}_1)$

and ignore others

Why is that?

To show it in code enter image description here

$\endgroup$
10
  • 1
    $\begingroup$ Please don't delete and re-post closed questions. Doing so destroys all of the work that others put into clarifying comments. Instead, edit the closed question to clarify what it is that you'd like to know more about. In this case, the answer to your question is contained in my comment to your previous post: the loss for a single sample is a sum, so it's entirely consistent to return a single loss value for a single instance. $\endgroup$
    – Sycorax
    Jul 14, 2021 at 16:14
  • $\begingroup$ No now I only have one data sample. I mean I have 3 output nodes. So the cross-entropy should calculate the loss for all three nodes then sum them up. But it only calculates for the 0th node's loss because the label is 0. $\endgroup$ Jul 14, 2021 at 16:20
  • $\begingroup$ i mean, z is not three data samples, it is one data sample. z is the output node's logits $\endgroup$ Jul 14, 2021 at 16:21
  • $\begingroup$ Re-read what I wrote. The loss of a single sample is the sum of the three nodes' predictions. But the expression $\sum_j y_j \log p_j$ has only one nonzero term, because only one element of $y$ is nonzero. $\endgroup$
    – Sycorax
    Jul 14, 2021 at 16:21
  • 2
    $\begingroup$ For a three label problem, the expression you've written isn't the correct loss, so perhaps that's the misunderstanding. $\endgroup$
    – Sycorax
    Jul 14, 2021 at 16:26

1 Answer 1

1
$\begingroup$

The loss of a single sample is the sum of the $J$ nodes' predictions, so for each sample, the loss is a scalar value.

The expression $-\sum_j^J y_j \log p_j$ has only one nonzero term, because only one element of $y$ is nonzero. This is the correct loss for a classification problem; we can prove that this is correct by writing out the likelihood of the categorical distribution and taking the negative of its logarithm to get the cross-entropy. It's merely a special case that we can re-write the binary classification problem in terms of one probability $p$ and one binary label $y$ in the convenient form $-y \log p - (1-y)\log (1-p)$.

Let's write out a concrete example. We have predictions $[0.4,0.3,0.2]$ and label $[1,0,0]$. By inspection, we know that the loss is $-\log(0.4)$ because this is the only entry with a nonzero label.

$\endgroup$
1
  • $\begingroup$ thanks! yea i have tried writing out the likelihood and take negative log to it and get the cross-entropy. I now am more confident it is correct. I think i was confused by multilabel classification (that I can use BCE to have the answer in my question) and multiclass classification (the correct one you just stated). Thank you very much $\endgroup$ Jul 14, 2021 at 16:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.