What loss function for multi-class, multi-label classification tasks in neural networks?

I'm training a neural network to classify a set of objects into n-classes. Each object can belong to multiple classes at the same time (multi-class, multi-label).

I read that for multi-class problems it is generally recommended to use softmax and categorical cross entropy as the loss function instead of mse and I understand more or less why.

For my problem of multi-label it wouldn't make sense to use softmax of course as each class probability should be independent from the other. So my final layer is just sigmoid units that squash their inputs into a probability range 0..1 for every class.

Now I'm not sure what loss function I should use for this. Looking at the definition of categorical crossentropy I believe it would not apply well to this problem as it will only take into account the output of neurons that should be 1 and ignores the others.

Binary cross entropy sounds like it would fit better, but I only see it ever mentioned for binary classification problems with a single output neuron.

I'm using python and keras for training in case it matters.

• I believe softmax is "sigmoid units that squash their inputs into a probability range 0..1 for every class". – Hong Ooi Sep 13 '16 at 9:46
• You can use softmax as your loss function and then use probabilities to multilabel your data. – balboa Sep 4 '17 at 12:25

If you are using keras, just put sigmoids on your output layer and binary_crossentropy on your cost function.

If you are using tensorflow, then can use sigmoid_cross_entropy_with_logits. But for my case this direct loss function was not converging. So I ended up using explicit sigmoid cross entropy loss $(y \cdot \ln(\text{sigmoid}(\text{logits})) + (1-y) \cdot \ln(1-\text{sigmoid}(\text{logits})))$ . You can make your own like in this Example

Sigmoid, unlike softmax don't give probability distribution around $n_{classes}$ as output, but independent probabilities.

If on average any row is assigned less labels then you can use softmax_cross_entropy_with_logits because with this loss while the classes are mutually exclusive, their probabilities need not be. All that is required is that each row of labels is a valid probability distribution. If they are not, the computation of the gradient will be incorrect.

• Dear Alok, can you explain to the OP how they would go about using this function and why it makes sense? As you will see in the tour, link only answers are not encouraged on the site. – Antoine Vernet Sep 12 '16 at 15:56
• A nice short explanation can be seen in keras github: github.com/fchollet/keras/issues/741 – Dror Hilman Mar 18 '17 at 5:43
• It's not recommended to write your own cost function when using cross entropy - it can be subject to numeric stability issues. See github.com/tensorflow/tensorflow/issues/2462 for a discussion. – kbrose Jun 15 '17 at 17:32
• One thing is multilabel, another thing is multilabel multiclass. Sigmoid squashes your output between 0 and 1, but the OP has multiple classes, so outputs should be E.g. 0 - 10. So outputs should look: [0,5,2,3,1] <--- this is not what sigmoid does. – mimoralea Nov 8 '17 at 17:41
• should i tf.round(logits) before using in cost function or can i directly use logits from hidden layer to tf.nn.sigmoid.... ? – Monk May 11 '18 at 17:22

UPDATE (18/04/18): The old answer still proved to be useful on my model. The trick is to model the partition function and the distribution separately, thus exploiting the power of softmax.

Consider your observation vector $y$ to contain $m$ labels. $y_{im}=\delta_{im}$ (1 if sample i contains label m, 0 otherwise). So the objective would be to to model the matrix in a per-sample manner. Hence the model evaluates $F(y_i,x_i)=-\log P(y_i|x_i)$. Consider expanding $y_{im}=Z\cdot P(y_m)$ to achieve two property:

1. Distribution function: $\sum_m P(y_m) = 1$
2. Partition function: $Z$ estimates the number of labels

Then it's a matter of modeling the two separately. The distribution function is best modeled with a softmax layer, and the partition function can be modeled with a linear unit (in practice I clipped it as $max(0.01,output)$. More sophisticated modeling like Poisson unit would probably work better). Then you can choose to apply distributed loss (KL on distribution and MSE on partition), or you can try the following loss on their product.

In practical, the choice of optimiser also makes a huge difference. My experience with the factorisation approach is it works best under Adadelta (Adagrad dont work for me, didnt try RMSprop yet, performances of SGD is subject to parameter).

Side comment on sigmoid: I have certainly tried sigmoid + crossentropy and it did not work out. The model inclined to predict the $Z$ only, and failed to capture the variation in distribution function. (aka, it's somehow quite useful for modelling the partition and there may be math reason behind it)

UPDATE: (Random thought) It seems using Dirichlet process would allow incorporation of some prior on the number of labels?

UPDATE: By experiment, the modified KL-divergence is still inclined to give multi-class output rather than multi-label output.

My experience with sigmoid cross-entropy was not very pleasant. At the moment I am using a modified KL-divergence. It takes the form

\begin{aligned} Loss(P,Q)&=\sum_x{|P(x)-Q(x)| \cdot \left|\log\frac{P(x)}{Q(x)}\right| } \\ &= \sum_x{\left| (P(x)-Q(x)) \cdot \log\frac{P(x)}{Q(x)}\right| } \end{aligned} Where $P(x)$ is the target pseudo-distribution and $Q(x)$ is the predicted pseudo-distribution (but the function is actually symmetrical so it does not actually matter)

They are called pseudo-distributions for not being normalised. So you can have $\sum_x{P(x)}=2$ if you have 2 labels for a particular sample.

Keras impelmentation

def abs_KL_div(y_true, y_pred):
y_true = K.clip(y_true, K.epsilon(), None)
y_pred = K.clip(y_pred, K.epsilon(), None)
return K.sum( K.abs( (y_true- y_pred) * (K.log(y_true / y_pred))), axis=-1)
• on my particular dataset, adam was much better than rmsprop – shadi Apr 9 '18 at 10:42
• If you use such loss for training, how to do it in testing phase? Also use softmax for the prediction, but how to select the threshold to determine multi-label classes? – karl_TUM Sep 17 at 9:18

I haven't used keras yet. Taking caffe for example, you can use SigmoidCrossEntropyLossLayer for multi-label problems.

• Care to explain why that's a good approach? – Firebug Oct 21 '16 at 15:47

Actually in tensorsflow you can still use the sigmoid_cross_entropy_mean as the loss calculation function in multi-label, I am very confirm it

• Give us a link to the documentation – Ivelin May 11 '17 at 17:11

I'm a newbie here but I'll try give it a shot with this question. I was searching the same thing as you, and finally I found a very good keras multi-class classification tutorial @ http://machinelearningmastery.com/multi-class-classification-tutorial-keras-deep-learning-library/.

The author of that tutorial use categorical cross entropy loss function, and there is other thread that may help you to find solution @ here.

• It's not only multi class , It's also multi labels. – Monk May 11 '18 at 3:37