I have been working on a multilabel classification problem. I want to classify whether each of 25 labels is present on a given sample. The labels are not mutually exclusive. Ultimately, I would like to rank the classifier's outputs to say something like "labels A, B, and D are most likely with probabilities X, Y, Z".

I have built a multioutput classifier using logistic regression as the base classifier in scikit. It looks like each label classifier is an independent binary classifier. My question is, how can I compare the probabilities output by each classifier? As I said, I ultimately want to be able to compare the likelihood of a given label with that of the other labels in order to rank the certainty of their appearing. I know logistic regression outputs well-calibrated models, but are the probabilities of the 25 binary classifiers directly comparable? Would calibrating these classifiers help to ensure that their output probabilities are comparable?

  • $\begingroup$ Are the labels independent of each other? $\endgroup$
    – Janosch
    Commented Jun 12, 2020 at 9:22
  • $\begingroup$ No. The covariance matrix is not diagonal $\endgroup$
    – Matt C
    Commented Jun 12, 2020 at 12:50

1 Answer 1


I can't see any reason why they wouldn't be comparable, but I don't know what I could cite to convince a skeptic of this. They're probabilities. As long as P(L present) + P(L not present) == 1 for all labels L (which they have to be to be probabilities), the probabilities of different labels should be comparable.

Nevertheless, there may be cases where, for instance, labels A and B both have probability greater than any other label, but the two are mutually exclusive, so if label A is present then label B is guaranteed to be absent, and vice-versa. These kinds of correlations will not be reflected in their probabilities. But that's OK, because, well, multidimensional distributions have means and also covariance matrices, and those are different.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.