# Is it okay to use cross entropy loss function with soft labels?

I have a classification problem where pixels will be labeled with soft labels (which denote probabilities) rather than hard 0,1 labels. Earlier with hard 0,1 pixel labeling the cross entropy loss function (sigmoidCross entropyLossLayer from Caffe) was giving decent results. Is it okay to use the sigmoid cross entropy loss layer (from Caffe) for this soft classification problem?

• I am looking for a cross entropy function which can deal with real-valued labels. Please let me know if you know an answer here
– Amir
May 31, 2016 at 2:10

The answer is yes, but you have to define it the right way.

Cross entropy is defined on probability distributions, not on single values. For discrete distributions $p$ and $q$, it's: $$H(p, q) = -\sum_y p(y) \log q(y)$$

When the cross entropy loss is used with 'hard' class labels, what this really amounts to is treating $p$ as the conditional empirical distribution over class labels. This is a distribution where the probability is 1 for the observed class label and 0 for all others. $q$ is the conditional distribution (probability of class label, given input) learned by the classifier. For a single observed data point with input $x_0$ and class $y_0$, we can see that the expression above reduces to the standard log loss (which would be averaged over all data points):

$$-\sum_y I\{y = y_0\} \log q(y \mid x_0) = -\log q(y_0 \mid x_0)$$

Here, $I\{\cdot\}$ is the indicator function, which is 1 when its argument is true or 0 otherwise (this is what the empirical distribution is doing). The sum is taken over the set of possible class labels.

In the case of 'soft' labels like you mention, the labels are no longer class identities themselves, but probabilities over two possible classes. Because of this, you can't use the standard expression for the log loss. But, the concept of cross entropy still applies. In fact, it seems even more natural in this case.

Let's call the class $y$, which can be 0 or 1. And, let's say that the soft label $s(x)$ gives the probability that the class is 1 (given the corresponding input $x$). So, the soft label defines a probability distribution:

$$p(y \mid x) = \left \{ \begin{array}{cl} s(x) & \text{If } y = 1 \\ 1-s(x) & \text{If } y = 0 \end{array} \right .$$

The classifier also gives a distribution over classes, given the input:

$$q(y \mid x) = \left \{ \begin{array}{cl} c(x) & \text{If } y = 1 \\ 1-c(x) & \text{If } y = 0 \end{array} \right .$$

Here, $c(x)$ is the classifier's estimated probability that the class is 1, given input $x$.

The task is now to determine how different these two distributions are, using the cross entropy. Plug these expressions for $p$ and $q$ into the definition of cross entropy, above. The sum is taken over the set of possible classes $\{0, 1\}$:

$$\begin{array}{ccl} H(p, q) & = & - p(y=0 \mid x) \log q(y=0 \mid x) - p(y=1 \mid x) \log q(y=1 \mid x)\\ & = & -(1-s(x)) \log (1-c(x)) - s(x) \log c(x) \end{array}$$

That's the expression for a single, observed data point. The loss function would be the mean over all data points. Of course, this can be generalized to multiclass classification as well.

• I keep coming back to the lucidity of this answer.
– auro
Jan 22, 2017 at 16:33
• "This can be generalized to multiclass classification as well": Do you mean by treating the labels as independent RV? In that case, p is again defined by N parameters (N is number of labels), except unlike for single-label example, they don't need to sum up to 1. Or do you mean without assuming independence? In that case we'd have a (hugely complex) distribution over all possible label sets, which in general would require $2^N$ parameters -- so of course, we'd only allow certain highly restricted distributions, representable by whatever family of low-parameter functions we choose.
– max
Sep 14, 2020 at 4:20
• @max the statement refers to multiclass classification, where each point is a member of one of $k$ mutually exclusive classes (generalizing binary classification where $k=2$). In this case, we simply have a categorical distribution over classes. It sounds like you're thinking of multilabel classification, where each point may simultaneously be a member of multiple classes (not mutually exclusive). The approach could be generalized to multilabel classification as well. But, as you've noted, the distribution over class labels requires more careful consideration in this case. Sep 14, 2020 at 5:46
• Opps yes sorry, didn't read carefully and was asking about multi-label classification :) Agreed, multi-class is a straightforward generalization. I guess multi-label case is very different, so would need a separate question to discuss.
– max
Sep 14, 2020 at 13:17
• @user20160: Thank you very much for the answer. One follow up: I know that the standard log loss can be easily derived from maximum likelihood estimation using hard labels. Can we also derive the cross entropy with soft labels via maximum likelihood estimation? Oct 1, 2020 at 23:30