4
$\begingroup$

I understand that for multi-class classification the correct loss to use is categorical cross-entropy. However, when performing mixup as a regularisation technique two samples $(X_1, y_1)$ and $(X_2, y_2)$ are combined to create a new sample such that $(X_{new}, y_{new}) = \lambda(X_1, y_1) + (1-\lambda)(X_2, y_2)$, which effectively gives the new sample two labels with different weights.

My question is should I be using categorical cross-entropy because we are classifying non-mixed samples during evaluation, or should I be using binary cross-entropy because the training has effectively become a multi-label classification problem?

Edit: Just to clarify this is a multi-class classification problem where all 100 classes are mutually exclusive, however during training mixup can cause a sample to be labelled with 2 classes where class $i$ has label weight $\lambda$ and class $j$ has label weight $1 -\lambda$. The two losses I am comparing are specifically keras.losses.BinaryCrossentropy and keras.losses.CategoricalCrossentropy. During evaluation, samples can only be labelled with one class.

Improve this question
$\endgroup$
10
  • 1
    $\begingroup$ The new sample is a convex combination of the two inputs. If the input labels match, then the label is either 0 or 1. If the labels don’t match, then the label is either $\lambda$ or $1-\lambda$. In any of these four cases, the BCE loss works because it achieves a minimum when the model predicts the correct label exactly — regardless of whether the label is 0,1 or in between. $\endgroup$
    – Sycorax ♦
    Commented Jun 29, 2021 at 17:48
  • $\begingroup$ @Sycorax perfect explanation, thank you! Additionally, should the output layer be using sigmoid activation as opposed to softmax? On one hand sigmoid is the 'standard' for multi-label with BCE, however I feel softmax may be more suited since the sample labels will always sum to exactly 1. $\endgroup$
    – Avelina
    Commented Jun 29, 2021 at 18:00
  • $\begingroup$ Both sum to 1. For a binary outcome, we can write $P(A) + P(A^c)=P(y=1)+P(y=0)=1$. For binary events, the difference wrt to outputs between sigmoid and softmax is that a sigmoid output solely gives $P(A)=P(y=1)$, while a softmax output gives $P(y=0), P(y=1)$. More broadly, you can show that for 2 classes, sigmoid is a special case of softmax. $\endgroup$
    – Sycorax ♦
    Commented Jun 29, 2021 at 19:24
  • $\begingroup$ @Sycorax yes I completely understand that for the 2 class case, however I have 100 classes, not just 2. $\endgroup$
    – Avelina
    Commented Jun 29, 2021 at 21:20
  • 1
    $\begingroup$ For instance, this concept is developed in the context of pixel intensities here stats.stackexchange.com/questions/206925/… and here stats.stackexchange.com/questions/490062/… $\endgroup$
    – Sycorax ♦
    Commented Jun 30, 2021 at 21:48

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.