# What loss function for multi-class, multi-label classification tasks in neural networks?

I'm training a neural network to classify a set of objects into n-classes. Each object can belong to multiple classes at the same time (multi-class, multi-label).

I read that for multi-class problems it is generally recommended to use softmax and categorical cross entropy as the loss function instead of mse and I understand more or less why.

For my problem of multi-label it wouldn't make sense to use softmax of course as each class probability should be independent from the other. So my final layer is just sigmoid units that squash their inputs into a probability range 0..1 for every class.

Now I'm not sure what loss function I should use for this. Looking at the definition of categorical crossentropy I believe it would not apply well to this problem as it will only take into account the output of neurons that should be 1 and ignores the others.

Binary cross entropy sounds like it would fit better, but I only see it ever mentioned for binary classification problems with a single output neuron.

I'm using python and keras for training in case it matters.

• I believe softmax is "sigmoid units that squash their inputs into a probability range 0..1 for every class". Sep 13 '16 at 9:46
• You can use softmax as your loss function and then use probabilities to multilabel your data. Sep 4 '17 at 12:25

If you are using keras, just put sigmoids on your output layer and binary_crossentropy on your cost function.

If you are using tensorflow, then can use sigmoid_cross_entropy_with_logits. But for my case this direct loss function was not converging. So I ended up using explicit sigmoid cross entropy loss $(y \cdot \ln(\text{sigmoid}(\text{logits})) + (1-y) \cdot \ln(1-\text{sigmoid}(\text{logits})))$ . You can make your own like in this Example

Sigmoid, unlike softmax don't give probability distribution around $n_{classes}$ as output, but independent probabilities.

If on average any row is assigned less labels then you can use softmax_cross_entropy_with_logits because with this loss while the classes are mutually exclusive, their probabilities need not be. All that is required is that each row of labels is a valid probability distribution. If they are not, the computation of the gradient will be incorrect.

• Dear Alok, can you explain to the OP how they would go about using this function and why it makes sense? As you will see in the tour, link only answers are not encouraged on the site. Sep 12 '16 at 15:56
• A nice short explanation can be seen in keras github: github.com/fchollet/keras/issues/741 Mar 18 '17 at 5:43
• It's not recommended to write your own cost function when using cross entropy - it can be subject to numeric stability issues. See github.com/tensorflow/tensorflow/issues/2462 for a discussion. Jun 15 '17 at 17:32
• One thing is multilabel, another thing is multilabel multiclass. Sigmoid squashes your output between 0 and 1, but the OP has multiple classes, so outputs should be E.g. 0 - 10. So outputs should look: [0,5,2,3,1] <--- this is not what sigmoid does. Nov 8 '17 at 17:41
• should i tf.round(logits) before using in cost function or can i directly use logits from hidden layer to tf.nn.sigmoid.... ? May 11 '18 at 17:22

UPDATE (18/04/18): The old answer still proved to be useful on my model. The trick is to model the partition function and the distribution separately, thus exploiting the power of softmax.

Consider your observation vector $y$ to contain $m$ labels. $y_{im}=\delta_{im}$ (1 if sample i contains label m, 0 otherwise). So the objective would be to to model the matrix in a per-sample manner. Hence the model evaluates $F(y_i,x_i)=-\log P(y_i|x_i)$. Consider expanding $y_{im}=Z\cdot P(y_m)$ to achieve two property:

1. Distribution function: $\sum_m P(y_m) = 1$
2. Partition function: $Z$ estimates the number of labels

Then it's a matter of modeling the two separately. The distribution function is best modeled with a softmax layer, and the partition function can be modeled with a linear unit (in practice I clipped it as $max(0.01,output)$. More sophisticated modeling like Poisson unit would probably work better). Then you can choose to apply distributed loss (KL on distribution and MSE on partition), or you can try the following loss on their product.

In practical, the choice of optimiser also makes a huge difference. My experience with the factorisation approach is it works best under Adadelta (Adagrad dont work for me, didnt try RMSprop yet, performances of SGD is subject to parameter).

Side comment on sigmoid: I have certainly tried sigmoid + crossentropy and it did not work out. The model inclined to predict the $Z$ only, and failed to capture the variation in distribution function. (aka, it's somehow quite useful for modelling the partition and there may be math reason behind it)

UPDATE: (Random thought) It seems using Dirichlet process would allow incorporation of some prior on the number of labels?

UPDATE: By experiment, the modified KL-divergence is still inclined to give multi-class output rather than multi-label output.

My experience with sigmoid cross-entropy was not very pleasant. At the moment I am using a modified KL-divergence. It takes the form

\begin{aligned} Loss(P,Q)&=\sum_x{|P(x)-Q(x)| \cdot \left|\log\frac{P(x)}{Q(x)}\right| } \\ &= \sum_x{\left| (P(x)-Q(x)) \cdot \log\frac{P(x)}{Q(x)}\right| } \end{aligned} Where $P(x)$ is the target pseudo-distribution and $Q(x)$ is the predicted pseudo-distribution (but the function is actually symmetrical so it does not actually matter)

They are called pseudo-distributions for not being normalised. So you can have $\sum_x{P(x)}=2$ if you have 2 labels for a particular sample.

Keras impelmentation

def abs_KL_div(y_true, y_pred):
y_true = K.clip(y_true, K.epsilon(), None)
y_pred = K.clip(y_pred, K.epsilon(), None)
return K.sum( K.abs( (y_true- y_pred) * (K.log(y_true / y_pred))), axis=-1)

• on my particular dataset, adam was much better than rmsprop Apr 9 '18 at 10:42
• If you use such loss for training, how to do it in testing phase? Also use softmax for the prediction, but how to select the threshold to determine multi-label classes? Sep 17 '19 at 9:18
• This updated answer makes no sense to me. There is no point in modelling the partition function if you already have normalized probabilities. As you stated, Z = 1 in that case. Mar 31 '21 at 4:30
• @user3180 Partition function here means "how many entities are there in my sample?". Probability means "what are these entities likely to be?". Partition says well there are 3 fruits in my image, and proba says these 3 things likely to be banana and apple, though not sure which one is which fruit. This is surely not the best approach, but at least an elephant model... Aug 3 '21 at 3:34

I was going through same problem, After some research here is my solution:

If you are using tensorflow :

Multi label loss:

    cross_entropy = tf.nn.sigmoid_cross_entropy_with_logits(logits=logits, labels=tf.cast(targets,tf.float32))

loss       = tf.reduce_mean(tf.reduce_sum(cross_entropy, axis=1))

prediction = tf.sigmoid(logits)
output     = tf.cast(self.prediction > threshold, tf.int32)


Explanation :

For example if Logits from model and labels are :

logits = array([[ 1.4397182 , -0.7993438 ,  4.113389  ,  3.2199187 ,  4.5777845 ],
[ 0.30619335,  0.10168511,  4.253479  ,  2.3782277 ,  4.7390924 ],
[ 1.124632  ,  1.6056736 ,  2.9778094 ,  2.0808482 ,  2.0735667 ],
[ 0.7051575 , -0.10341895,  4.990803  ,  3.7019827 ,  3.8265839 ],
[ 0.6333333 , -0.76601076,  3.2255085 ,  2.7842572 ,  5.3817415 ]],
dtype=float32)

labels = array([[1, 1, 0, 0, 0],
[0, 1, 0, 0, 1],
[1, 1, 1, 1, 0],
[0, 0, 1, 0, 1],
[1, 1, 1, 1, 1]])

then

cross_entropy = tf.nn.sigmoid_cross_entropy_with_logits(logits=logits, labels=tf.cast(targets,tf.float32))

will give you :

[[0.21268466 1.170648   4.129609   3.2590992  4.58801   ]
[0.85791767 0.64359653 4.2675934  2.466893   0.00870855]
[0.28124034 0.18294993 0.04965096 0.11762683 2.1920042 ]
[1.1066352  0.64277405 0.00677719 3.7263577  0.02155003]
[0.42580318 1.147773   0.03896642 0.059942   0.00458926]]

and

prediction = tf.cast(tf.sigmoid(one_placeholder) > 0.5, tf.int32)

will give you :

[[1 0 1 1 1]
[1 1 1 1 1]
[1 1 1 1 1]
[1 0 1 1 1]
[1 0 1 1 1]]


Now you have predicted labels and true labels, You can calculate accuracy easily.

For multi-class:

The labels must be one-hot encoded

cross_entropy = tf.nn.softmax_cross_entropy_with_logits_v2(logits=logits, labels = one_hot_y)
loss = tf.reduce_sum(cross_entropy)

predictions = tf.argmax(logits, axis=1, output_type=tf.int32, name='predictions')
accuracy = tf.reduce_sum(tf.cast(tf.equal(predictions, true_labels), tf.float32))


Another example

# LOSS AND OPTIMIZER
loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(logits=output, labels=y))
beta1=0.9,
beta2=0.999,
epsilon=1e-08).minimize(loss, global_step=global_step)

# PREDICTION AND ACCURACY CALCULATION
correct_prediction = tf.equal(y_pred_cls, tf.argmax(y, axis=1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))


I haven't used keras yet. Taking caffe for example, you can use SigmoidCrossEntropyLossLayer for multi-label problems.

• Care to explain why that's a good approach? Oct 21 '16 at 15:47

Actually in tensorsflow you can still use the sigmoid_cross_entropy_mean as the loss calculation function in multi-label, I am very confirm it

I'm a newbie here but I'll try give it a shot with this question. I was searching the same thing as you, and finally I found a very good keras multi-class classification tutorial @ http://machinelearningmastery.com/multi-class-classification-tutorial-keras-deep-learning-library/.

The author of that tutorial use categorical cross entropy loss function, and there is other thread that may help you to find solution @ here.

• It's not only multi class , It's also multi labels. May 11 '18 at 3:37