Activation function when there are several output labels

Question

If we had a NN to, let's say, clasify images of digits, but each image could contain more than 1 digit (all different), is there any problem in using a 10-dimensional output layer (representing digits 0-9) and the sigmoid activation function in each output neuron?

We cannot use softmax here because, as I say, more than 1 outputs are possible. E.g. if an image contains the digits 3, 5 and 9, the ground truth vector would be [0001010001].

I read some people suggested duplicating the output. I.e: if we can have up to 5 different numbers in an image, we would have 10x5=50 output neurons and apply a softmax activation function to each group of 10. I think this approach is wrong.

I appreciate any other ideas!

Answer 1

This is sometimes called multi-label classification.

Assuming you have no constraints on the number of labels – e.g. if there are two labels, each of 00, 01, 10, and 11 is possible – your strategy of a separate sigmoid activation for each label is probably the default neural network approach. You could view this as training a separate classifier for each label, but with all of the intermediate hidden representations shared between the classifiers.

I read some people suggested duplicating the output. I.e: if we can have up to 5 different numbers in an image, we would have 10x5=50 output neurons and apply a softmax activation function to each group of 10. I think this approach is wrong.

This approach would indeed be strange; assuming the labels aren't ordered, you'd have to make your loss do a "matching": predictions of (1, 3) and (3, 1) need to be treated the same. Even if you did that somehow, it seems unlikely that this approach would be better than your suggestion, except maybe if you know e.g. that only three of the ten possible labels can be "turned on" for any given instance.