89
$\begingroup$

I am playing with convolutional neural networks using Keras+Tensorflow to classify categorical data. I have a choice of two loss functions: categorial_crossentropy and sparse_categorial_crossentropy.

I have a good intuition about the categorial_crossentropy loss function, which is defined as follows:

$$ J(\textbf{w}) = -\frac{1}{N} \sum_{i=1}^{N} \left[ y_i \text{log}(\hat{y}_i) + (1-y_i) \text{log}(1-\hat{y}_i) \right] $$

where,

  • $\textbf{w}$ refer to the model parameters, e.g. weights of the neural network
  • $y_i$ is the true label
  • $\hat{y_i}$ is the predicted label

Both labels use the one-hot encoded scheme.

Questions:

  • How does the above loss function change in sparse_categorial_crossentropy?
  • What is the mathematical intuition behind it?
  • When to use one over the other?
Improve this question
$\endgroup$
3
  • $\begingroup$ Does stackoverflow.com/questions/37312421/… answer your questions? $\endgroup$
    – Mark L. Stone
    May 10, 2018 at 17:32
  • $\begingroup$ @MarkL.Stone it answers the question partially. I am looking for a mathematical intuition as to how sparsity affects the cost function. $\endgroup$
    – kedarps
    May 11, 2018 at 14:03
  • $\begingroup$ @kedarps they are mathematically identical, although sparse CE has the restriction that the labels $y_i$ are hard (0 or 1). Also depending on the implementation, sparse CE is possibly cheaper in terms of computation. $\endgroup$
    – shimao
    May 12, 2018 at 16:08

5 Answers 5

Reset to default
106
$\begingroup$

Both, categorical cross entropy and sparse categorical cross entropy have the same loss function which you have mentioned above. The only difference is the format in which you mention $Y_i$ (i,e true labels).

If your $Y_i$'s are one-hot encoded, use categorical_crossentropy. Examples (for a 3-class classification): [1,0,0] , [0,1,0], [0,0,1]

But if your $Y_i$'s are integers, use sparse_categorical_crossentropy. Examples for above 3-class classification problem: [1] , [2], [3]

The usage entirely depends on how you load your dataset. One advantage of using sparse categorical cross entropy is it saves time in memory as well as computation because it simply uses a single integer for a class, rather than a whole vector.

Improve this answer
$\endgroup$
4
  • 3
    $\begingroup$ What does the sparse refer to in sparse categorical cross-entropy? I thought it was because the data was sparsely distributed among the classes. $\endgroup$
    – nid
    May 19, 2020 at 11:44
  • 2
    $\begingroup$ it sparse because of using 10 values to store one correct class (in case of mnist), it uses only one value . $\endgroup$
    – Amit Portnoy
    Jun 29, 2020 at 18:21
  • 2
    $\begingroup$ It seems it is more than just mater of data format take a look at this $\endgroup$
    – Ali Asgari
    Mar 11, 2021 at 5:57
  • $\begingroup$ Can't we use simple categorical_crossentropy if Yi's are integers? $\endgroup$
    – Thunder
    Apr 23, 2021 at 2:59
12
$\begingroup$

The formula which you posted in your question refers to binary_crossentropy, not categorical_crossentropy. The former is used when you have only one class. The latter refers to a situation when you have multiple classes and its formula looks like below:

$$J(\textbf{w}) = -\sum_{i=1}^{N} y_i \text{log}(\hat{y}_i).$$

This loss works as skadaver mentioned on one-hot encoded values e.g [1,0,0], [0,1,0], [0,0,1]

The sparse_categorical_crossentropy is a little bit different, it works on integers that's true, but these integers must be the class indices, not actual values. This loss computes logarithm only for output index which ground truth indicates to. So when model output is for example [0.1, 0.3, 0.7] and ground truth is 3 (if indexed from 1) then loss compute only logarithm of 0.7. This doesn't change the final value, because in the regular version of categorical crossentropy other values are immediately multiplied by zero (because of one-hot encoding characteristic). Thanks to that it computes logarithm once per instance and omits the summation which leads to better performance. The formula might look like this:

$$J(\textbf{w}) = -\text{log}(\hat{y}_y).$$

Improve this answer
$\endgroup$
7
$\begingroup$

I have no better answer than the links and me too encountered the same question. I just want to point out, that the formula for loss function (cross entropy) seems to be a little bit erroneous (and might be misleading.) One should probably drop the 2nd term in the bracket to have simply $$J(\textbf{w}) = -\frac{1}{N} \sum_{i=1}^{N} y_i \text{log}(\hat{y}_i).$$ Sorry for writing my comment here, but I haven't got enough reputation points to be able to comment...

Improve this answer
$\endgroup$
2
  • $\begingroup$ This is the standard technical definition of entropy, but I believe it's not commonly used as a loss function because it's not symmetric between 0-1 labels. In fact, if the true y_i is 0, this would calculate the loss to also be zero, regardless of prediction. OP's version corrects for this symmetry. $\endgroup$
    – Joey F.
    Nov 22, 2018 at 5:23
  • 1
    $\begingroup$ @JoeyF. Yes, I do notice that when I read the original post. There seems to be quite some discussion on the Internet about this issue e.g.. I tried to conceive an illustrative example in a short time but failed. What others said about the gradient involves also the label-0 $\hat{y}_i$. However, I still believe those who named it with cross entropy did it with a reason and the fact that since the $\hat{y}_i$ sums to $1$, maximizing any one of them automatically minimizes the rest. $\endgroup$
    – phunc20
    Nov 22, 2018 at 7:02
3
$\begingroup$

By the nature of your question, it sounds like you have 3 or more categories. However, for the sake of completion I would like to add that if you are dealing with a binary classification, using binary cross entropy might be more appropriate.

Furthermore, be careful to choose the loss and metric properly, since this can lead to some unexpected and weird behaviour in the performance of your model.

Improve this answer
$\endgroup$
0
$\begingroup$

This explanation is taken from Andrew Ng's Coursera course "Machine Learning Specialization":

SparseCategoricalCrossentropy: Expects the target to be an int corresponding to the index. For example, if there are 10 potential target values, y would be between 0 and 9. CategoricalCrossentropy: Expects the target value of an example to be one-hot encoded where the value at the target index is 1, which the other N-1 entries are 0. An example with 10 potential target values, where the target is 2 would be [0,0,1,0,0,0,0,0,0,0].

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.