35
$\begingroup$

First of all, I realized if I need to perform binary predictions, I have to create at least two classes through performing a one-hot-encoding. Is this correct? However, is binary cross entropy only for predictions with only one class? If I were to use a categorical cross entropy loss which is typically found in most libraries (like TensorFlow), would there be a significant difference?

In fact, what are the exact differences between a categorical and binary cross entropy? I have never seen an implementation of binary cross entropy in TensorFlow so I thought perhaps the categorical one works just as fine.

$\endgroup$
50
$\begingroup$

Binomial cross-entropy loss is a special case of multinomial cross-entropy loss for $m=2$.

$$ \begin{align} \mathcal{L}(\theta) &= -\frac{1}{n}\sum_{i=1}^n \left[y_i \log(p_i) + (1-y_i) \log(1-p_i)\right] \\ &= -\frac{1}{n}\sum_{i=1}^n\sum_{j=1}^m y_{ij} \log(p_{ij}) \end{align} $$

Where $i$ indexes samples/observations and $j$ indexes classes, and $y$ is the sample label (binary for LSH, one-hot vector on the RHS) and $p_{ij}\in(0,1):\sum_{j} p_{ij} =1\forall i,j$ is the prediction for a sample.

$\endgroup$
  • 3
    $\begingroup$ Does it mean to say so long as I use 2 classes in a multinomial cross entropy loss, I am essentially using a binary cross entropy loss? $\endgroup$ – infomin101 Feb 7 '17 at 17:40
  • 2
    $\begingroup$ @leekwotsin yup $\endgroup$ – Reinstate Monica Feb 7 '17 at 17:57
17
$\begingroup$

Binary cross-entropy is for multi-label classifications, whereas categorical cross entropy is for multi-class classification where each example belongs to a single class.

$\endgroup$

Some of the information contained in this post requires additional references. Please edit to add citations to reliable sources that support the assertions made here. Unsourced material may be disputed or deleted.

  • 3
    $\begingroup$ What is the justification for your statement? Why wouldn't you use categorical cross entropy to multi-label classification? $\endgroup$ – michal Feb 5 '18 at 14:44
  • $\begingroup$ what if there are multiple labels, each containing multiple classes? $\endgroup$ – slizb Feb 7 '18 at 13:34
  • 1
    $\begingroup$ This is what exactly I wanted to hear, but not what my boss wants to hear. A little bit of explanation would have been so awesome. $\endgroup$ – Aditya Sep 17 at 9:43
2
$\begingroup$

I think there are three kinds of classification tasks:

  1. Binary classification: two exclusive classes
  2. Multi-class classification: more than two exclusive classes
  3. Multi-label classification: just non-exclusive classes

From these, we can say

  • In the case of (1), you need to use binary cross entropy.
  • In the case of (2), you need to use categorical cross entropy.
  • In the case of (3), you need to use binary cross entropy. You can just consider the multi-label classifier as a multi separate binary classifier. If you have 10 classes here, you have 10 binary classifiers separately. Each binary classifier is trained independently. Thus, we can produce multi-label for each sample. If you want to make sure at least one label must be acquired, then you can select the one with the lowest classification loss function, or using other metrics.

I want to emphasize that multi-class classification is not similar to multi-label classification! Rather, multi-label classifier borrows an idea from the binary classifier!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.