I am training a neural network for multilabel classification, with a large number of classes (1000). Which means more than one output can be active for every input. On an average, I have two classes active per output frame. On training with a cross entropy loss the neural network resorts to outputting only zeros, because it gets the least loss with this output since 99.8% of my labels are zeros. Any suggestions on how I can push the network to give more weight to the positive classes?

  • $\begingroup$ What are you using as software? Python + Keras? $\endgroup$ Feb 10 '17 at 14:49
  • $\begingroup$ Btw: 99.8% is just a number, you know that a 0.2% of error on average corresponds to 0.002*1000, so 2 wrong labels per training instance on average. BTW are you using categorical cross_entropy or binary_crossentropy with sigmoids on the last layer? $\endgroup$ Feb 10 '17 at 14:52
  • $\begingroup$ @TommasoGuerrini used python+ keras, sigmoid and binary_crossentropy. Now testing with categorical_crossentropy, the network is outputting values closer to 1 now. But the loss is too high for now. Waiting to see how it trains over more epochs now. $\endgroup$
    – Yakku
    Feb 10 '17 at 15:14
  • $\begingroup$ @TommasoGuerrini I did not understand the purpose of the callback. $\endgroup$
    – Yakku
    Feb 10 '17 at 15:27
  • $\begingroup$ my bad, just an example on which value of loss makes sense $\endgroup$ Feb 10 '17 at 15:29

Tensorflow has a loss function weighted_cross_entropy_with_logits, which can be used to give more weight to the 1's. So it should be applicable to a sparse multi-label classification setting like yours.

From the documentation:

This is like sigmoid_cross_entropy_with_logits() except that pos_weight, allows one to trade off recall and precision by up- or down-weighting the cost of a positive error relative to a negative error.

The argument pos_weight is used as a multiplier for the positive targets

If you use the tensorflow backend in Keras, you can use the loss function like this (Keras 2.1.1):

import tensorflow as tf
import keras.backend.tensorflow_backend as tfb

POS_WEIGHT = 10  # multiplier for positive targets, needs to be tuned

def weighted_binary_crossentropy(target, output):
    Weighted binary crossentropy between an output tensor 
    and a target tensor. POS_WEIGHT is used as a multiplier 
    for the positive targets.

    Combination of the following functions:
    * keras.losses.binary_crossentropy
    * keras.backend.tensorflow_backend.binary_crossentropy
    * tf.nn.weighted_cross_entropy_with_logits
    # transform back to logits
    _epsilon = tfb._to_tensor(tfb.epsilon(), output.dtype.base_dtype)
    output = tf.clip_by_value(output, _epsilon, 1 - _epsilon)
    output = tf.log(output / (1 - output))
    # compute weighted loss
    loss = tf.nn.weighted_cross_entropy_with_logits(targets=target,
    return tf.reduce_mean(loss, axis=-1)

Then in your model:

model.compile(loss=weighted_binary_crossentropy, ...)

I have not found many resources yet which report well working values for the pos_weight in relation to the number of classes, average active classes, etc.

  • 1
    $\begingroup$ Also, it might be a good idea to evaluate the f-measure in a callback after each epoch when tuning the hyperparameters (such as pos_weights). $\endgroup$
    – tobigue
    Nov 15 '17 at 16:56
  • 1
    $\begingroup$ Is there a corresponding weighted_binary_accuracy metric that can be used for the model as well? $\endgroup$ Oct 21 '19 at 8:20
  • $\begingroup$ Lifesaver, but I could also use something like weighted_binary_accuracy $\endgroup$
    – David Cian
    Jun 16 '20 at 17:26
  • $\begingroup$ You can just use binary accuracy actually, unless you really want to weigh the accuracy as well $\endgroup$
    – David Cian
    Jun 16 '20 at 17:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.