57
$\begingroup$

First of all, I realized if I need to perform binary predictions, I have to create at least two classes through performing a one-hot-encoding. Is this correct? However, is binary cross-entropy only for predictions with only one class? If I were to use a categorical cross-entropy loss, which is typically found in most libraries (like TensorFlow), would there be a significant difference?

In fact, what are the exact differences between a categorical and binary cross-entropy? I have never seen an implementation of binary cross-entropy in TensorFlow, so I thought perhaps the categorical one works just as fine.

$\endgroup$
2
69
$\begingroup$

Bernoulli$^*$ cross-entropy loss is a special case of categorical cross-entropy loss for $m=2$.

$$ \begin{align} \mathcal{L}(\theta) &= -\frac{1}{n}\sum_{i=1}^n \left[y_i \log(p_i) + (1-y_i) \log(1-p_i)\right] \\ &= -\frac{1}{n}\sum_{i=1}^n\sum_{j=1}^m y_{ij} \log(p_{ij}) \end{align} $$

Where $i$ indexes samples/observations and $j$ indexes classes, and $y$ is the sample label (binary for LSH, one-hot vector on the RHS) and $p_{ij}\in(0,1):\sum_{j} p_{ij} =1\forall i,j$ is the prediction for a sample.


I write "Bernoulli cross-entropy" because this loss arises from a Bernoulli probability model. There is not a "binary distribution." A "binary cross-entropy" doesn't tell us if the thing that is binary is the one-hot vector of $k \ge 2$ labels, or if the author is using binary encoding for each trial (success or failure). This isn't a general convention, but it makes clear that these formulae arise from particular probability models. Conventional jargon is not clear in that way.

$\endgroup$
7
  • 5
    $\begingroup$ Does it mean to say so long as I use 2 classes in a multinomial cross entropy loss, I am essentially using a binary cross entropy loss? $\endgroup$ – infomin101 Feb 7 '17 at 17:40
  • 2
    $\begingroup$ @leekwotsin yup $\endgroup$ – Sycorax Feb 7 '17 at 17:57
  • $\begingroup$ If this is the minus log likelihood, then you don't need the $1/n$ term. Not sure why it is there, because it plays no role in minimising $\mathcal{L}$. $\endgroup$ – Dmitry Zotikov Oct 28 '20 at 19:45
  • 1
    $\begingroup$ @DmitryZotikov It's true that a positive rescaling does not change the location of the optima. At the same time, it's very common to characterize neural network loss functions in terms of averages because changing the mini-batch size and using a sum implicitly changes the step size of gradient-based training. On the other hand, an average de-couples mini-batch size and learning rate. See: stats.stackexchange.com/questions/358786/… $\endgroup$ – Sycorax Oct 28 '20 at 19:49
  • 1
    $\begingroup$ It's an estimate of the cross-entropy of the model probability and the empirical probability in the data, which is the expected negative log probability according to the model averaged across the data. Hence the names categorical/binary cross entropy loss :) $\endgroup$ – jkpate Oct 29 '20 at 14:18
34
$\begingroup$

There are three kinds of classification tasks:

  1. Binary classification: two exclusive classes
  2. Multi-class classification: more than two exclusive classes
  3. Multi-label classification: just non-exclusive classes

Here, we can say

  • In the case of (1), you need to use binary cross entropy.
  • In the case of (2), you need to use categorical cross entropy.
  • In the case of (3), you need to use binary cross entropy.

You can just consider the multi-label classifier as a combination of multiple independent binary classifiers. If you have 10 classes here, you have 10 binary classifiers separately. Each binary classifier is trained independently. Thus, we can produce multi-label for each sample. If you want to make sure at least one label must be acquired, then you can select the one with the lowest classification loss function, or using other metrics.

I want to emphasize that multi-class classification is not similar to multi-label classification! Rather, multi-label classifier borrows an idea from the binary classifier!

$\endgroup$
1
  • $\begingroup$ I understand your point. However, couldn't we use categorical cross-entropy in each of the 3 cases? In both (1) and (3), categorical cross-entropy with 2 classes could be used, and I don't see any difference with using binary cross-entropy (they just coincide as functions!) $\endgroup$ – Abramodj Nov 11 '20 at 8:56
22
$\begingroup$

Binary cross-entropy is for multi-label classifications, whereas categorical cross entropy is for multi-class classification where each example belongs to a single class.

$\endgroup$
6
  • 5
    $\begingroup$ What is the justification for your statement? Why wouldn't you use categorical cross entropy to multi-label classification? $\endgroup$ – michal Feb 5 '18 at 14:44
  • $\begingroup$ what if there are multiple labels, each containing multiple classes? $\endgroup$ – slizb Feb 7 '18 at 13:34
  • 4
    $\begingroup$ This is what exactly I wanted to hear, but not what my boss wants to hear. A little bit of explanation would have been so awesome. $\endgroup$ – Aditya Sep 17 '19 at 9:43
  • 3
    $\begingroup$ this answer should be down-voted as it lacks of follow-up clarification $\endgroup$ – Long Feb 12 '20 at 3:55
  • 1
    $\begingroup$ @michal CCE can't really be used for multi-label classification as it only outputs one "thing" as the output. We would need several "things" classified in multi-label classification, hence we need multiple sigmoid outputs. $\endgroup$ – Eoin Ó Coinnigh Jan 4 at 17:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.