Cross Entropy vs. Sparse Cross Entropy: When to use one over the other I am playing with convolutional neural networks using Keras+Tensorflow to classify categorical data. I have a choice of two loss functions: categorial_crossentropy and sparse_categorial_crossentropy.
I have a good intuition about the categorial_crossentropy loss function, which is defined as follows:
$$
J(\textbf{w}) = -\frac{1}{N} \sum_{i=1}^{N} \left[ y_i \text{log}(\hat{y}_i) + (1-y_i) \text{log}(1-\hat{y}_i) \right]
$$ 
where,


*

*$\textbf{w}$ refer to the model parameters, e.g. weights of the neural network

*$y_i$ is the true label

*$\hat{y_i}$ is the predicted label


Both labels use the one-hot encoded scheme. 
Questions:


*

*How does the above loss function change in sparse_categorial_crossentropy?

*What is the mathematical intuition behind it?

*When to use one over the other?

 A: Both, categorical cross entropy and sparse categorical cross entropy have the same loss function which you have mentioned above.
The only difference is the format in which you mention $Y_i$ (i,e true labels).
If your $Y_i$'s are one-hot encoded, use categorical_crossentropy.
Examples (for a 3-class classification):  [1,0,0] , [0,1,0], [0,0,1]
But if your $Y_i$'s are integers, use sparse_categorical_crossentropy.
Examples for above 3-class classification problem: 
[1] , [2], [3]
The usage entirely depends on how you load your dataset. One advantage of using sparse categorical cross entropy is it saves time in memory as well as computation because it simply uses a single integer for a class, rather than a whole vector.
A: The formula which you posted in your question refers to binary_crossentropy, not categorical_crossentropy. The former is used when you have only one class. The latter refers to a situation when you have multiple classes and its formula looks like below:
$$J(\textbf{w}) = -\sum_{i=1}^{N} y_i \text{log}(\hat{y}_i).$$
This loss works as skadaver mentioned on one-hot encoded values e.g [1,0,0], [0,1,0], [0,0,1]
The sparse_categorical_crossentropy is a little bit different, it works on integers that's true, but these integers must be the class indices, not actual values. This loss computes logarithm only for output index which ground truth indicates to. So when model output is for example [0.1, 0.3, 0.7] and ground truth is 3 (if indexed from 1) then loss compute only logarithm of 0.7. This doesn't change the final value, because in the regular version of categorical crossentropy other values are immediately multiplied by zero (because of one-hot encoding characteristic). Thanks to that it computes logarithm once per instance and omits the summation which leads to better performance. The formula might look like this:
$$J(\textbf{w}) = -\text{log}(\hat{y}_y).$$
A: I have no better answer than the links and me too encountered the same question. I just want to point out, that the formula for loss function (cross entropy) seems to be a little bit erroneous (and might be misleading.) One should probably drop the 2nd term in the bracket to have simply $$J(\textbf{w}) = -\frac{1}{N} \sum_{i=1}^{N} y_i \text{log}(\hat{y}_i).$$
Sorry for writing my comment here, but I haven't got enough reputation points to be able to comment...
A: By the nature of your question, it sounds like you have 3 or more categories. However, for the sake of completion I would like to add that if you are dealing with a binary classification, using binary cross entropy might be more appropriate. 
Furthermore, be careful to choose the loss and metric properly, since this can lead to some unexpected and weird behaviour in the performance of your model.
