# Cross Entropy vs. Sparse Cross Entropy: When to use one over the other

I am playing with convolutional neural networks using Keras+Tensorflow to classify categorical data. I have a choice of two loss functions: categorial_crossentropy and sparse_categorial_crossentropy.

I have a good intuition about the categorial_crossentropy loss function, which is defined as follows:

$$J(\textbf{w}) = -\frac{1}{N} \sum_{i=1}^{N} \left[ y_i \text{log}(\hat{y}_i) + (1-y_i) \text{log}(1-\hat{y}_i) \right]$$

where,

• $\textbf{w}$ refer to the model parameters, e.g. weights of the neural network
• $y_i$ is the true label
• $\hat{y_i}$ is the predicted label

Both labels use the one-hot encoded scheme.

### Questions:

• How does the above loss function change in sparse_categorial_crossentropy?
• What is the mathematical intuition behind it?
• When to use one over the other?
• Does stackoverflow.com/questions/37312421/… answer your questions? – Mark L. Stone May 10 '18 at 17:32
• @MarkL.Stone it answers the question partially. I am looking for a mathematical intuition as to how sparsity affects the cost function. – kedarps May 11 '18 at 14:03
• @kedarps they are mathematically identical, although sparse CE has the restriction that the labels $y_i$ are hard (0 or 1). Also depending on the implementation, sparse CE is possibly cheaper in terms of computation. – shimao May 12 '18 at 16:08

Both, categorical cross entropy and sparse categorical cross entropy have the same loss function which you have mentioned above. The only difference is the format in which you mention $$Y_i$$ (i,e true labels).

If your $$Y_i$$'s are one-hot encoded, use categorical_crossentropy. Examples (for a 3-class classification): [1,0,0] , [0,1,0], [0,0,1]

But if your $$Y_i$$'s are integers, use sparse_categorical_crossentropy. Examples for above 3-class classification problem: [1] , [2], [3]

The usage entirely depends on how you load your dataset. One advantage of using sparse categorical cross entropy is it saves time in memory as well as computation because it simply uses a single integer for a class, rather than a whole vector.

• What does the sparse refer to in sparse categorical cross-entropy? I thought it was because the data was sparsely distributed among the classes. – Nicolas May 19 at 11:44

I have no better answer than the links and me too encountered the same question. I just want to point out, that the formula for loss function (cross entropy) seems to be a little bit erroneous (and might be misleading.) One should probably drop the 2nd term in the bracket to have simply $$J(\textbf{w}) = -\frac{1}{N} \sum_{i=1}^{N} y_i \text{log}(\hat{y}_i).$$ Sorry for writing my comment here, but I haven't got enough reputation points to be able to comment...

• This is the standard technical definition of entropy, but I believe it's not commonly used as a loss function because it's not symmetric between 0-1 labels. In fact, if the true y_i is 0, this would calculate the loss to also be zero, regardless of prediction. OP's version corrects for this symmetry. – Joey F. Nov 22 '18 at 5:23
• @JoeyF. Yes, I do notice that when I read the original post. There seems to be quite some discussion on the Internet about this issue e.g.. I tried to conceive an illustrative example in a short time but failed. What others said about the gradient involves also the label-0 $\hat{y}_i$. However, I still believe those who named it with cross entropy did it with a reason and the fact that since the $\hat{y}_i$ sums to $1$, maximizing any one of them automatically minimizes the rest. – wucf20 Nov 22 '18 at 7:02

By the nature of your question, it sounds like you have 3 or more categories. However, for the sake of completion I would like to add that if you are dealing with a binary classification, using binary cross entropy might be more appropriate.

Furthermore, be careful to choose the loss and metric properly, since this can lead to some unexpected and weird behaviour in the performance of your model.