Theoretical justification for training a multi-class classification model to be used for multi-label classification Can a multi-class classification model be trained and used for multi-label classification, under any mathematical-theoretical guarantee?
Imagine the following model, actually used in one machine learning library for (text) classification:


*

*A multi-class classifier ― a softmax terminated MLP feeding from word embeddings, but could be anything else as well ― is trained on multi-label data. (i.e. some/most data items have multiple class designations in the training data).

*The loss per training item is computed while accounting for only a single target label, selected at random at each epoch, from among the labels applying to the item in the training data (exact loss function here, excuse the C++). This is just a small speed-inspired variant to standard stochastic gradient descent... which should average out over epochs.

*For actual usage, the confidence threshold which maximizes the aggregate Jaccard index over the entire test set, is then used for filtering the labels returned as the network's (softmax normalized) prediction output.

*For each prediction made by the model, only those labels that have confidence larger than the threshold, are kept and considered as the final actionable prediction.


This may feel like a coercion of a multi-class model into a multi-label interpretation. Are there any theoretical guarantees, or counter-guarantees, to this being useful for multilabel semantics? or, how would you reduce a multilabel problem to this?
 A: A softmax output layer does not seem to make sense. The total probability of all classes would then be coerced to sum to 1. This does not make sense in a multi-label setting. Using a sigmoid instead would seem more logical (allows multiple classes to have high probability e.g. close to 1). 
Perhaps what is being done in steps 2, 3 and 4 is a consequence of having to compensate for having used a softmax activiation function for the output layer? At least the bit in step 2 ensures that this matches up with a softmax activiation. I am not so sure that this evens out over epochs: each category will occur less frequently in the created training data than is really the case in the training data and you throw away all information on what categories tend to occur together (unless you have so little data that there is a concern about overfitting to that?!). Additionally, I assume you would get better performance (=not speed-wise, but from the prediction perspective) if you did have the multiple labels in each step. I find it hard to believe that this really cost that much in speed and I would assume that it would improve your predictions to use all the labels at all times.
In short, I see some reasons for why what you describe could go wrong and assume (without really knowing) that some of the contortions in the approach try to compensate for some of these. I do not know enough and have not tried this, so I cannot say how successful this would be. I conjecture that the correlation between the multiple labels is something that this approach would not capture.
Personally, I would be tempted to just do this as a proper multi-label prediction with a final dense layer with a number of units equal to the number of classes (i.e. encoded as e.g. 1 0 0 0 1 0 0 1 0 0 ... if an item falls into the 1st, 4th and 8th class) and a sigmoid activation (using e.g. binary crossentropy as the loss function). I believe this is the standard approach normally recommended for this situation.
