In machine learning, cross-entropy is often used while training a neural network.

During my training of my neural network, I track the accuracy and the cross entropy. The accuracy is pretty low, so I know that my network isn't performing well. But what can I say about my model knowing the cross-entropy?


Andrew Ng explains the intuition behind using cross-entropy as a cost function in his ML Coursera course under the logistic regression module, specifically at this point in time with the mathematical expression:

$$\text{Cost}\left(h_\theta(x),y\right)=\left\{ \begin{array}{l} -\log\left(h_\theta(x)\right) \quad \quad\quad \text{if $y =1$}\\ -\log\left(1 -h_\theta(x)\right) \quad \;\text{if $y =0$} \end{array} \right. $$

The idea is that with an activation function with values between zero and one (in this case a logistic sigmoid, but clearly applicable to, for instance, a softmax function in CNN, where the final output is a multinomial logistic), the cost in the case of a true 1 value ($y=1$), will decrease from infinity to zero as $h_\theta(x)\to1$, because ideally we would like for its to be $1$, predicting exactly the true value, and hence rewarding an activation output that gets close to it; reciprocally, the cost will tend to infinity as the activation function tends to $0$. The opposite is true for $y=0$ with the trick of obtaining the logarithm of $1-h_\theta(x)$, as opposed to $h_\theta(x).$

Here is my attempt at showing this graphically, as we limit these two functions between the vertical lines at $0$ and $1$, consistent with the output of a sigmoid function:

enter image description here

This can be summarized in one more succinct expression as:


In the case of softmax in CNN, the cross-entropy would similarly be formulated as

$$\text{Cost}=-\sum_j \,t_j\,\log(y_j)$$

where $t_j$ stands for the target value of each class, and $y_j$ the probability assigned to it by the output.

Beyond the intuition, the introduction of cross entropy is meant to make the cost function convex.

  • 2
    $\begingroup$ To the beyond intuition part I would also add that cross entropy also emerges from maximum likelihood estimation for logistic regression model $\endgroup$ – Łukasz Grad Apr 11 '17 at 21:34
  • $\begingroup$ Is this only for binary case? .. what about above? $\endgroup$ – Bob Burt Apr 30 '17 at 3:30
  • $\begingroup$ @BobBurt I included a link to a page explaining the extrapolation to softmax and the cross entropy equations that follow, $\endgroup$ – Antoni Parellada Apr 30 '17 at 6:28
  • $\begingroup$ @BobBurt This answer explains the relationship between binary and multinomial cross-entropy. stats.stackexchange.com/questions/260505/… $\endgroup$ – Sycorax Nov 16 '18 at 0:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.