2
$\begingroup$

This article provides a nice explanation and derivation of cross entropy, and defines it as follows:

$H(y, \hat{y}) = -\sum_i[ y_i \log \hat{y}_i]$ ....... (1)

where $y$ is the correct output and $\hat{y}$ is the network's predicted output.

Many neural network articles use this definition of cross entropy:

$Loss = -\sum_i[ y_i \log \hat{y}_i + (1-y_i) \log(1- \hat{y}_i))]$ ....... (2)

These equations are different, where does equation 2 come from? Is equation 2 a better loss function for neural networks?

$\endgroup$

marked as duplicate by Reinstate Monica, mdewey, jld, kjetil b halvorsen, John Aug 8 '17 at 16:58

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

2
$\begingroup$

If $i \in \{1,0\}$ these equations are identical. The second is commonly used for the case of a binary response whereas the first is a more general form used for $k$ classes.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.