Should I use a categorical cross-entropy or binary cross-entropy loss for binary predictions? First of all, I realized if I need to perform binary predictions, I have to create at least two classes through performing a one-hot-encoding. Is this correct? However, is binary cross-entropy only for predictions with only one class? If I were to use a categorical cross-entropy loss, which is typically found in most libraries (like TensorFlow), would there be a significant difference?
In fact, what are the exact differences between a categorical and binary cross-entropy? I have never seen an implementation of binary cross-entropy in TensorFlow, so I thought perhaps the categorical one works just as fine.
 A: Bernoulli$^*$ cross-entropy loss is a special case of categorical cross-entropy loss for $m=2$.
$$
\begin{align}
\mathcal{L}(\theta)
&= -\frac{1}{n}\sum_{i=1}^n\sum_{j=1}^m y_{ij}\log(p_{ij}) \\
&= -\frac{1}{n}\sum_{i=1}^n \left[y_i \log(p_i) + (1-y_i) \log(1-p_i)\right] 
\end{align}
$$
Where $i$ indexes samples/observations and $j$ indexes classes, and $y$ is the sample label (binary for LSH, one-hot vector on the RHS) and $p_{ij}\in(0,1):\sum_{j} p_{ij} =1\forall i,j$ is the prediction for a sample.

I write "Bernoulli cross-entropy" because this loss arises from a Bernoulli probability model. There is not a "binary distribution." A "binary cross-entropy" doesn't tell us if the thing that is binary is the one-hot vector of $k \ge 2$ labels, or if the author is using binary encoding for each trial (success or failure). This isn't a general convention, but it makes clear that these formulae arise from particular probability models. Conventional jargon is not clear in that way.
A: There are three kinds of classification tasks:  


*

*Binary classification: two exclusive classes    

*Multi-class classification: more than two exclusive classes

*Multi-label classification: just non-exclusive classes


Here, we can say


*

*In the case of (1), you need to use binary cross entropy.  

*In the case of (2), you need to use categorical cross entropy.  

*In the case of (3), you need to use binary cross entropy. 


You can just consider the multi-label classifier as a combination of multiple independent binary classifiers. If you have 10 classes here, you have 10 binary classifiers separately. Each binary classifier is trained independently. Thus, we can produce multi-label for each sample. If you want to make sure at least one label must be acquired, then you can select the one with the lowest classification loss function, or using other metrics.
I want to emphasize that multi-class classification is not similar to multi-label classification! Rather, multi-label classifier borrows an idea from the binary classifier!
A: Binary cross-entropy is for multi-label classifications, whereas categorical cross entropy is for multi-class classification where each example belongs to a single class.
A: Binary Cross Entropy is a special case of Categorical Cross Entropy with 2 classes (class=1, and class=0).  If we formulate Binary Cross Entropy this way, then we can use the general Cross-Entropy loss formula here: Sum(y*log y) for each class.  Notice how this is the same as binary cross entropy.
For multi-label classification, the idea is the same.  But instead of say 3 labels to indicate 3 classes, we have 6 labels to indicate presence or absence of each class (class1=1, class1=0, class2=1, class2=0, class3=1, and class3=0).  The loss then is the sum of cross-entropy loss for each of these 6 classes.
