Can a multi-class classification model be trained and used for multi-label classification, under any mathematical-theoretical guarantee?

Imagine the following model, actually used in one machine learning library for (text) classification:

  1. A multi-class classifier ― a softmax terminated MLP feeding from word embeddings, but could be anything else as well ― is trained on multi-label data. (i.e. some/most data items have multiple class designations in the training data).
  2. The loss per training item is computed while accounting for only a single target label, selected at random at each epoch, from among the labels applying to the item in the training data (exact loss function here, excuse the C++). This is just a small speed-inspired variant to standard stochastic gradient descent... which should average out over epochs.
  3. For actual usage, the confidence threshold which maximizes the aggregate Jaccard index over the entire test set, is then used for filtering the labels returned as the network's (softmax normalized) prediction output.
  4. For each prediction made by the model, only those labels that have confidence larger than the threshold, are kept and considered as the final actionable prediction.

This may feel like a coercion of a multi-class model into a multi-label interpretation. Are there any theoretical guarantees, or counter-guarantees, to this being useful for multilabel semantics? or, how would you reduce a multilabel problem to this?

  • $\begingroup$ I mean that for example, results will bear a meaning not saying which single class is most likely to apply (as in multiclass classification), but that multilabel semantics will apply to the output. New data being from a different distribution is a standard issue with machine learning models and has relatively nothing to do with this question. Thanks for the title edit though. $\endgroup$
    – matanox
    Commented Apr 4, 2018 at 7:20
  • $\begingroup$ Very interesting question, but very complicated to answer -- not that I would know an answer or where to find it. There are several helpful restrictions, but to me the abstraction of the determination threshold -- the decision rule whether to accept a label -- seems rather awkward to formalize. With only ANN in mind and considering that some of the "easy" proofs are already quite complicated, I'd be sceptical that you could prove any guarantee. I'm not familiar with text classification, but in my fields of application a different approach proved to be more successful: start with regression... $\endgroup$
    – cherub
    Commented Apr 6, 2018 at 9:13
  • $\begingroup$ ... so, instead of using classification, one tries to model the CDF. A theoretical proof is not yet given, but from general considerations it is clear, that the CDF holds all information. Regarding the training, it has the unfortunate exponential scaling behaviour wrt the number of dimensions. One remedy is to use different networks for each category, so you don't have to restrict the number of categories. Classification might then be done in a second step with any means that yield the best answer, evaluated in posterior. $\endgroup$
    – cherub
    Commented Apr 6, 2018 at 9:16
  • $\begingroup$ Sorry, I didn't fully grasp where this is going. Happy to learn more... $\endgroup$
    – matanox
    Commented Apr 10, 2018 at 10:51
  • $\begingroup$ What you described is in some aspects similar to the standard method, expensive but works very well. The theoretical judgement can be made by constructing a generative process and analyze the likelihood. Other methods as the one suggested in Bjorn's answer are faster but less accurate. $\endgroup$
    – THN
    Commented Aug 16, 2019 at 2:04

1 Answer 1


A softmax output layer does not seem to make sense. The total probability of all classes would then be coerced to sum to 1. This does not make sense in a multi-label setting. Using a sigmoid instead would seem more logical (allows multiple classes to have high probability e.g. close to 1).

Perhaps what is being done in steps 2, 3 and 4 is a consequence of having to compensate for having used a softmax activiation function for the output layer? At least the bit in step 2 ensures that this matches up with a softmax activiation. I am not so sure that this evens out over epochs: each category will occur less frequently in the created training data than is really the case in the training data and you throw away all information on what categories tend to occur together (unless you have so little data that there is a concern about overfitting to that?!). Additionally, I assume you would get better performance (=not speed-wise, but from the prediction perspective) if you did have the multiple labels in each step. I find it hard to believe that this really cost that much in speed and I would assume that it would improve your predictions to use all the labels at all times.

In short, I see some reasons for why what you describe could go wrong and assume (without really knowing) that some of the contortions in the approach try to compensate for some of these. I do not know enough and have not tried this, so I cannot say how successful this would be. I conjecture that the correlation between the multiple labels is something that this approach would not capture.

Personally, I would be tempted to just do this as a proper multi-label prediction with a final dense layer with a number of units equal to the number of classes (i.e. encoded as e.g. 1 0 0 0 1 0 0 1 0 0 ... if an item falls into the 1st, 4th and 8th class) and a sigmoid activation (using e.g. binary crossentropy as the loss function). I believe this is the standard approach normally recommended for this situation.

  • $\begingroup$ Many thanks for the great deliberation! Something small ― you mention "in the created training data", but the random choice is made during training, so I think it could possibly be rephrased or differently put... and I think in terms of the expectancy, it will be equivalent and hence the justification for that step ― which I agree is a mostly an undesirable tradeoff of speed with accuracy/performance ― is nonetheless rationalized. $\endgroup$
    – matanox
    Commented Apr 10, 2018 at 10:52
  • $\begingroup$ Additionally, to clarify the answer, the final layer is already fully connected to the hidden layer, and has a number of units equal to the number of classes.... or are you suggesting an additional layer, or am I mis-reading this? $\endgroup$
    – matanox
    Commented Apr 10, 2018 at 11:02
  • $\begingroup$ My point was that the true training data seems to have multiple labels per item, but the approach would not preserve that and instead generate training data that has one class present. This throws away all information on what classes occur together, even if some other adjustments somehow manage to get their overall frequencies right. $\endgroup$
    – Björn
    Commented Apr 10, 2018 at 16:47
  • $\begingroup$ The standard approach would not differ in terms of what is fully connected, but rather what training data is used (I.e. With multi -labelled instances) and what the activation function for the final layer is. $\endgroup$
    – Björn
    Commented Apr 10, 2018 at 16:49
  • $\begingroup$ I should apologize: a single label is selected at random per datum per epoch during the training. I've only now more explicitly reflected that in the question. In case this changes your perspective feel free updating the answer; we'd probably remove our comments here to clean up anyway I guess... $\endgroup$
    – matanox
    Commented Apr 10, 2018 at 17:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.