In machine learning, cross-entropy is often used while training a neural network.
During my training of my neural network, I track the accuracy and the cross entropy. The accuracy is pretty low, so I know that my network isn't performing well. But what can I say about my model knowing the cross-entropy?
Andrew Ng explains the intuition behind using cross-entropy as a cost function in his ML Coursera course under the logistic regression module, specifically at this point in time with the mathematical expression:
$$\text{Cost}\left(h_\theta(x),y\right)=\left\{ \begin{array}{l} -\log\left(h_\theta(x)\right) \quad \quad\quad \text{if $y =1$}\\ -\log\left(1 -h_\theta(x)\right) \quad \;\text{if $y =0$} \end{array} \right. $$
The idea is that with an activation function with values between zero and one (in this case a logistic sigmoid, but clearly applicable to, for instance, a softmax function in CNN, where the final output is a multinomial logistic), the cost in the case of a true 1 value ($y=1$), will decrease from infinity to zero as $h_\theta(x)\to1$, because ideally we would like for its to be $1$, predicting exactly the true value, and hence rewarding an activation output that gets close to it; reciprocally, the cost will tend to infinity as the activation function tends to $0$. The opposite is true for $y=0$ with the trick of obtaining the logarithm of $1-h_\theta(x)$, as opposed to $h_\theta(x).$
Here is my attempt at showing this graphically, as we limit these two functions between the vertical lines at $0$ and $1$, consistent with the output of a sigmoid function:
This can be summarized in one more succinct expression as:
$$\text{Cost}\left(h_\theta(x),y\right)=-y\log\left(h_\theta(x)\right)-(1-y) \log\left(1 - h_\theta(x)\right).$$
In the case of softmax in CNN, the cross-entropy would similarly be formulated as
$$\text{Cost}=-\sum_j \,t_j\,\log(y_j)$$
where $t_j$ stands for the target value of each class, and $y_j$ the probability assigned to it by the output.
Beyond the intuition, the introduction of cross entropy is meant to make the cost function convex.
