6
$\begingroup$

I am playing with convolutional neural networks using Keras+Tensorflow to classify categorical data. I have a choice of two loss functions: categorial_crossentropy and sparse_categorial_crossentropy.

I have a good intuition about the categorial_crossentropy loss function, which is defined as follows:

$$ J(\textbf{w}) = -\frac{1}{N} \sum_{i=1}^{N} \left[ y_i \text{log}(\hat{y}_i) + (1-y_i) \text{log}(1-\hat{y}_i) \right] $$

where,

  • $\textbf{w}$ refer to the model parameters, e.g. weights of the neural network
  • $y_i$ is the true label
  • $\hat{y_i}$ is the predicted label

Both labels use the one-hot encoded scheme.

Questions:

  • How does the above loss function change in sparse_categorial_crossentropy?
  • What is the mathematical intuition behind it?
  • When to use one over the other?
$\endgroup$
  • $\begingroup$ Does stackoverflow.com/questions/37312421/… answer your questions? $\endgroup$ – Mark L. Stone May 10 '18 at 17:32
  • $\begingroup$ @MarkL.Stone it answers the question partially. I am looking for a mathematical intuition as to how sparsity affects the cost function. $\endgroup$ – kedarps May 11 '18 at 14:03
  • $\begingroup$ @kedarps they are mathematically identical, although sparse CE has the restriction that the labels $y_i$ are hard (0 or 1). Also depending on the implementation, sparse CE is possibly cheaper in terms of computation. $\endgroup$ – shimao May 12 '18 at 16:08
5
$\begingroup$

Both, categorical cross entropy and sparse categorical cross entropy have the same loss function which you have mentioned above. The only difference is the format in which you mention Yi (i,e true labels).

If your Yi's are one-hot encoded, use categorical_crossentropy. Examples (for a 3-class classification): [1,0,0] , [0,1,0], [0,0,1]

But if your Yi's are integers, use sparse_categorical_crossentropy. Examples for above 3-class classification problem: [1] , [2], [3]

The usage entirely depends on how you load your dataset. One advantage of using sparse categorical cross entropy is it saves time in memory as well as computation because it simply uses a single integer for a class, rather than a whole vector.

$\endgroup$
2
$\begingroup$

I have no better answer than the links and me too encountered the same question. I just want to point out, that the formula for loss function (cross entropy) seems to be a little bit erroneous (and might be misleading.) One should probably drop the 2nd term in the bracket to have simply $$J(\textbf{w}) = -\frac{1}{N} \sum_{i=1}^{N} y_i \text{log}(\hat{y}_i).$$ Sorry for writing my comment here, but I haven't got enough reputation points to be able to comment...

$\endgroup$
  • $\begingroup$ This is the standard technical definition of entropy, but I believe it's not commonly used as a loss function because it's not symmetric between 0-1 labels. In fact, if the true y_i is 0, this would calculate the loss to also be zero, regardless of prediction. OP's version corrects for this symmetry. $\endgroup$ – Joey F. Nov 22 '18 at 5:23
  • $\begingroup$ @JoeyF. Yes, I do notice that when I read the original post. There seems to be quite some discussion on the Internet about this issue e.g.. I tried to conceive an illustrative example in a short time but failed. What others said about the gradient involves also the label-0 $\hat{y}_i$. However, I still believe those who named it with cross entropy did it with a reason and the fact that since the $\hat{y}_i$ sums to $1$, maximizing any one of them automatically minimizes the rest. $\endgroup$ – wucf20 Nov 22 '18 at 7:02
0
$\begingroup$

By the nature of your question, it sounds like you have 3 or more categories. However, for the sake of completion I would like to add that if you are dealing with a binary classification, using binary cross entropy might be more appropriate.

Furthermore, be careful to choose the loss and metric properly, since this can lead to some unexpected and weird behaviour in the performance of your model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.