Trying to train a CNN on a multilabel problem, each image can have 0, 1, 2 or 3 labels assigned to it. The number of labels is not known a priori. I figured the standard loss function for such problem is the sigmoid cross entropy (as opposed to the softmax cross entropy, which would only be appropriate for single class problems). For example, in TensorFlow one would use sigmoid_cross_entropy_with_logits.
I did that, but found my network to not converge much beyond some trivial level. Looking in more detail at the errors made, I found my network to perform well on examples with 0 or 1 labels while performing badly on examples with more than 1 label. In fact, there is a strong correlation between the number of labels in an example and the cross-entropy loss.
The picture below shows the cross entropy for 64 test samples, color coded for how many labels they were supposed to have. I consider everything with a loss of less than 0.3 a "pretty good" result. Clearly examples with 0 or 1 label are "pretty good" and more labels get progressively worse.
The question is, what can I do to improve my cost function? What can there be possibly wrong? It must be a fairly generic problem as many of these cat and dog classifiers allow multiple classes/labels.
The four categories (0,1,2,3 labels) are about equally represented in the training data. This related question does not answer beyond "use sigmoid cross entropy".
