What loss function for multi-class, multi-label classification tasks in neural networks?

Question

I'm training a neural network to classify a set of objects into n-classes. Each object can belong to multiple classes at the same time (multi-class, multi-label).

I read that for multi-class problems it is generally recommended to use softmax and categorical cross entropy as the loss function instead of mse and I understand more or less why.

For my problem of multi-label it wouldn't make sense to use softmax of course as each class probability should be independent from the other. So my final layer is just sigmoid units that squash their inputs into a probability range 0..1 for every class.

Now I'm not sure what loss function I should use for this. Looking at the definition of categorical crossentropy I believe it would not apply well to this problem as it will only take into account the output of neurons that should be 1 and ignores the others.

Binary cross entropy sounds like it would fit better, but I only see it ever mentioned for binary classification problems with a single output neuron.

I'm using python and keras for training in case it matters.

Answer 1

If you are using keras, just put sigmoids on your output layer and binary_crossentropy on your cost function.

If you are using tensorflow, then can use sigmoid_cross_entropy_with_logits. But for my case this direct loss function was not converging. So I ended up using explicit sigmoid cross entropy loss $(y \cdot \ln(\text{sigmoid}(\text{logits})) + (1-y) \cdot \ln(1-\text{sigmoid}(\text{logits})))$ . You can make your own like in this Example

Sigmoid, unlike softmax don't give probability distribution around $n_{classes}$ as output, but independent probabilities.

If on average any row is assigned less labels then you can use softmax_cross_entropy_with_logits because with this loss while the classes are mutually exclusive, their probabilities need not be. All that is required is that each row of labels is a valid probability distribution. If they are not, the computation of the gradient will be incorrect.

Answer 2

UPDATE (18/04/18): The old answer still proved to be useful on my model. The trick is to model the partition function and the distribution separately, thus exploiting the power of softmax.

Consider your observation vector $y$ to contain $m$ labels. $y_{im}=\delta_{im}$ (1 if sample i contains label m, 0 otherwise). So the objective would be to to model the matrix in a per-sample manner. Hence the model evaluates $F(y_i,x_i)=-\log P(y_i|x_i)$. Consider expanding $y_{im}=Z\cdot P(y_m)$ to achieve two property:

1. Distribution function: $\sum_m P(y_m) = 1$
2. Partition function: $Z$ estimates the number of labels

Then it's a matter of modeling the two separately. The distribution function is best modeled with a softmax layer, and the partition function can be modeled with a linear unit (in practice I clipped it as $max(0.01,output)$. More sophisticated modeling like Poisson unit would probably work better). Then you can choose to apply distributed loss (KL on distribution and MSE on partition), or you can try the following loss on their product.

In practical, the choice of optimiser also makes a huge difference. My experience with the factorisation approach is it works best under Adadelta (Adagrad dont work for me, didnt try RMSprop yet, performances of SGD is subject to parameter).

Side comment on sigmoid: I have certainly tried sigmoid + crossentropy and it did not work out. The model inclined to predict the $Z$ only, and failed to capture the variation in distribution function. (aka, it's somehow quite useful for modelling the partition and there may be math reason behind it)

UPDATE: (Random thought) It seems using Dirichlet process would allow incorporation of some prior on the number of labels?

UPDATE: By experiment, the modified KL-divergence is still inclined to give multi-class output rather than multi-label output.

---

(Old answer)

My experience with sigmoid cross-entropy was not very pleasant. At the moment I am using a modified KL-divergence. It takes the form

$$ \begin{aligned} Loss(P,Q)&=\sum_x{|P(x)-Q(x)| \cdot \left|\log\frac{P(x)}{Q(x)}\right| } \\ &= \sum_x{\left| (P(x)-Q(x)) \cdot \log\frac{P(x)}{Q(x)}\right| } \end{aligned} $$ Where $P(x)$ is the target pseudo-distribution and $Q(x)$ is the predicted pseudo-distribution (but the function is actually symmetrical so it does not actually matter)

They are called pseudo-distributions for not being normalised. So you can have $\sum_x{P(x)}=2$ if you have 2 labels for a particular sample.

Keras impelmentation

def abs_KL_div(y_true, y_pred):
    y_true = K.clip(y_true, K.epsilon(), None)
    y_pred = K.clip(y_pred, K.epsilon(), None)
    return K.sum( K.abs( (y_true- y_pred) * (K.log(y_true / y_pred))), axis=-1)

Answer 3

I was going through same problem, After some research here is my solution:

If you are using tensorflow :

Multi label loss:

    cross_entropy = tf.nn.sigmoid_cross_entropy_with_logits(logits=logits, labels=tf.cast(targets,tf.float32))

    loss       = tf.reduce_mean(tf.reduce_sum(cross_entropy, axis=1))

    prediction = tf.sigmoid(logits)
    output     = tf.cast(self.prediction > threshold, tf.int32)
    train_op   = tf.train.AdamOptimizer(0.001).minimize(loss)

Explanation :

For example if Logits from model and labels are :

logits = array([[ 1.4397182 , -0.7993438 ,  4.113389  ,  3.2199187 ,  4.5777845 ],
       [ 0.30619335,  0.10168511,  4.253479  ,  2.3782277 ,  4.7390924 ],
       [ 1.124632  ,  1.6056736 ,  2.9778094 ,  2.0808482 ,  2.0735667 ],
       [ 0.7051575 , -0.10341895,  4.990803  ,  3.7019827 ,  3.8265839 ],
       [ 0.6333333 , -0.76601076,  3.2255085 ,  2.7842572 ,  5.3817415 ]],
      dtype=float32)

labels = array([[1, 1, 0, 0, 0],
       [0, 1, 0, 0, 1],
       [1, 1, 1, 1, 0],
       [0, 0, 1, 0, 1],
       [1, 1, 1, 1, 1]])

then

cross_entropy = tf.nn.sigmoid_cross_entropy_with_logits(logits=logits, labels=tf.cast(targets,tf.float32))

will give you :

[[0.21268466 1.170648   4.129609   3.2590992  4.58801   ]
 [0.85791767 0.64359653 4.2675934  2.466893   0.00870855]
 [0.28124034 0.18294993 0.04965096 0.11762683 2.1920042 ]
 [1.1066352  0.64277405 0.00677719 3.7263577  0.02155003]
 [0.42580318 1.147773   0.03896642 0.059942   0.00458926]]

and

prediction = tf.cast(tf.sigmoid(one_placeholder) > 0.5, tf.int32) 

will give you :  

[[1 0 1 1 1]
 [1 1 1 1 1]
 [1 1 1 1 1]
 [1 0 1 1 1]
 [1 0 1 1 1]]

Now you have predicted labels and true labels, You can calculate accuracy easily.

For multi-class:

The labels must be one-hot encoded

cross_entropy = tf.nn.softmax_cross_entropy_with_logits_v2(logits=logits, labels = one_hot_y)
loss = tf.reduce_sum(cross_entropy)

optimizer = tf.train.AdamOptimizer(learning_rate=self.lr).minimize(loss)

predictions = tf.argmax(logits, axis=1, output_type=tf.int32, name='predictions')
accuracy = tf.reduce_sum(tf.cast(tf.equal(predictions, true_labels), tf.float32))

Another example

# LOSS AND OPTIMIZER
loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(logits=output, labels=y))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate,
                                   beta1=0.9,
                                   beta2=0.999,
                                   epsilon=1e-08).minimize(loss, global_step=global_step)


# PREDICTION AND ACCURACY CALCULATION
correct_prediction = tf.equal(y_pred_cls, tf.argmax(y, axis=1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

Answer 4

I haven't used keras yet. Taking caffe for example, you can use SigmoidCrossEntropyLossLayer for multi-label problems.

Answer 5

Actually in tensorsflow you can still use the sigmoid_cross_entropy_mean as the loss calculation function in multi-label, I am very confirm it

Answer 6

I'm a newbie here but I'll try give it a shot with this question. I was searching the same thing as you, and finally I found a very good keras multi-class classification tutorial @ http://machinelearningmastery.com/multi-class-classification-tutorial-keras-deep-learning-library/.

The author of that tutorial use categorical cross entropy loss function, and there is other thread that may help you to find solution @ here.