60
$\begingroup$

First of all, I realized if I need to perform binary predictions, I have to create at least two classes through performing a one-hot-encoding. Is this correct? However, is binary cross-entropy only for predictions with only one class? If I were to use a categorical cross-entropy loss, which is typically found in most libraries (like TensorFlow), would there be a significant difference?

In fact, what are the exact differences between a categorical and binary cross-entropy? I have never seen an implementation of binary cross-entropy in TensorFlow, so I thought perhaps the categorical one works just as fine.

$\endgroup$
2
73
$\begingroup$

Bernoulli$^*$ cross-entropy loss is a special case of categorical cross-entropy loss for $m=2$.

$$ \begin{align} \mathcal{L}(\theta) &= -\frac{1}{n}\sum_{i=1}^n\sum_{j=1}^m y_{ij}\log(p_{ij}) \\ &= -\frac{1}{n}\sum_{i=1}^n \left[y_i \log(p_i) + (1-y_i) \log(1-p_i)\right] \end{align} $$

Where $i$ indexes samples/observations and $j$ indexes classes, and $y$ is the sample label (binary for LSH, one-hot vector on the RHS) and $p_{ij}\in(0,1):\sum_{j} p_{ij} =1\forall i,j$ is the prediction for a sample.


I write "Bernoulli cross-entropy" because this loss arises from a Bernoulli probability model. There is not a "binary distribution." A "binary cross-entropy" doesn't tell us if the thing that is binary is the one-hot vector of $k \ge 2$ labels, or if the author is using binary encoding for each trial (success or failure). This isn't a general convention, but it makes clear that these formulae arise from particular probability models. Conventional jargon is not clear in that way.

$\endgroup$
7
  • 6
    $\begingroup$ Does it mean to say so long as I use 2 classes in a multinomial cross entropy loss, I am essentially using a binary cross entropy loss? $\endgroup$
    – infomin101
    Feb 7 '17 at 17:40
  • 3
    $\begingroup$ @leekwotsin yup $\endgroup$
    – Sycorax
    Feb 7 '17 at 17:57
  • 2
    $\begingroup$ @DmitryZotikov It's true that a positive rescaling does not change the location of the optima. At the same time, it's very common to characterize neural network loss functions in terms of averages because changing the mini-batch size and using a sum implicitly changes the step size of gradient-based training. On the other hand, an average de-couples mini-batch size and learning rate. See: stats.stackexchange.com/questions/358786/… $\endgroup$
    – Sycorax
    Oct 28 '20 at 19:49
  • 1
    $\begingroup$ It's an estimate of the cross-entropy of the model probability and the empirical probability in the data, which is the expected negative log probability according to the model averaged across the data. Hence the names categorical/binary cross entropy loss :) $\endgroup$
    – jkpate
    Oct 29 '20 at 14:18
  • 1
    $\begingroup$ @jkpate Can you point to a reference? $\endgroup$ Oct 29 '20 at 17:14
37
$\begingroup$

There are three kinds of classification tasks:

  1. Binary classification: two exclusive classes
  2. Multi-class classification: more than two exclusive classes
  3. Multi-label classification: just non-exclusive classes

Here, we can say

  • In the case of (1), you need to use binary cross entropy.
  • In the case of (2), you need to use categorical cross entropy.
  • In the case of (3), you need to use binary cross entropy.

You can just consider the multi-label classifier as a combination of multiple independent binary classifiers. If you have 10 classes here, you have 10 binary classifiers separately. Each binary classifier is trained independently. Thus, we can produce multi-label for each sample. If you want to make sure at least one label must be acquired, then you can select the one with the lowest classification loss function, or using other metrics.

I want to emphasize that multi-class classification is not similar to multi-label classification! Rather, multi-label classifier borrows an idea from the binary classifier!

$\endgroup$
1
  • $\begingroup$ I understand your point. However, couldn't we use categorical cross-entropy in each of the 3 cases? In both (1) and (3), categorical cross-entropy with 2 classes could be used, and I don't see any difference with using binary cross-entropy (they just coincide as functions!) $\endgroup$
    – Abramodj
    Nov 11 '20 at 8:56
22
$\begingroup$

Binary cross-entropy is for multi-label classifications, whereas categorical cross entropy is for multi-class classification where each example belongs to a single class.

$\endgroup$
6
  • 5
    $\begingroup$ What is the justification for your statement? Why wouldn't you use categorical cross entropy to multi-label classification? $\endgroup$
    – michal
    Feb 5 '18 at 14:44
  • $\begingroup$ what if there are multiple labels, each containing multiple classes? $\endgroup$
    – slizb
    Feb 7 '18 at 13:34
  • 4
    $\begingroup$ This is what exactly I wanted to hear, but not what my boss wants to hear. A little bit of explanation would have been so awesome. $\endgroup$
    – Aditya
    Sep 17 '19 at 9:43
  • 3
    $\begingroup$ this answer should be down-voted as it lacks of follow-up clarification $\endgroup$
    – Long
    Feb 12 '20 at 3:55
  • 1
    $\begingroup$ @michal CCE can't really be used for multi-label classification as it only outputs one "thing" as the output. We would need several "things" classified in multi-label classification, hence we need multiple sigmoid outputs. $\endgroup$ Jan 4 at 17:16
0
$\begingroup$

Binary Cross Entropy is a special case of Categorical Cross Entropy with 2 classes (class=1, and class=0). If we formulate Binary Cross Entropy this way, then we can use the general Cross-Entropy loss formula here: Sum(y*log y) for each class. Notice how this is the same as binary cross entropy.

For multi-label classification, the idea is the same. But instead of say 3 labels to indicate 3 classes, we have 6 labels to indicate presence or absence of each class (class1=1, class1=0, class2=1, class2=0, class3=1, and class3=0). The loss then is the sum of cross-entropy loss for each of these 6 classes.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.