2
$\begingroup$

There seems to be a gap in the literature as to why cross-entropy is used.

Older references on neural networks ("ANNs") always use the squared loss. For example, here is one from Chong and Zak "An Intro to Optimization 4th Ed",

enter image description here

Here is the one by Simon Haykin on "Kalman Filter and Neural Networks" enter image description here


Somewhere along the way, cross-entropy became the dominant loss function that is used in many papers and almost all the "blog" type references on NN. Recall that the cross-entropy is often formulated as,

$$ CE(y, \hat y) = -\sum\limits_{n = 1}^N \sum\limits_{c = 1}^n y_n^c \cdot\log(\hat y_n^c) $$ where $n$ is the $n$th data, $c$ is the $c$th class, and $y, \hat y$ denotes the set of targets and the predictions, respectively.

Where did the above function even came from (books/papers)? Was there some famous work that used cross entropy that popularized it over the squared loss? Is there a good reason to use CE as opposed to the squared loss (or the softmax loss associated with softmax/multiclass logistic regression)?

$\endgroup$
4
  • $\begingroup$ Do you require a neural network to have 1 or more hidden layers? $\endgroup$
    – Sycorax
    Apr 1, 2020 at 19:55
  • $\begingroup$ @SycoraxsaysReinstateMonica Not necessarily. One layer is good in my opinion. $\endgroup$
    – Fraïssé
    Apr 1, 2020 at 20:03
  • $\begingroup$ What about 0 layers? $\endgroup$
    – Sycorax
    Apr 1, 2020 at 20:15
  • $\begingroup$ @SycoraxsaysReinstateMonica Fine by me $\endgroup$
    – Fraïssé
    Apr 1, 2020 at 21:51

1 Answer 1

2
$\begingroup$

If you agree that a logistic regression is a special case of a neural network, then the answer is D. R. Cox, who invented logistic regression.[1]

A neural network with zero hidden layers and a single sigmoid output and trained to maximize the binomial likelihood (equiv. minimize cross-entropy) is logistic regression.

Minimizing a binomial cross-entropy is equivalent to maximizing a particular likelihood: the relationship between maximizing the likelihood and minimizing the cross-entropy

[1] D. R. Cox. "The Regression Analysis of Binary Sequences" Journal of the Royal Statistical Society. Series B (Methodological) Vol. 20, No. 2 (1958), pp. 215-242.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.