Ranking most probable labels from multilabel classifier

Question

I have been working on a multilabel classification problem. I want to classify whether each of 25 labels is present on a given sample. The labels are not mutually exclusive. Ultimately, I would like to rank the classifier's outputs to say something like "labels A, B, and D are most likely with probabilities X, Y, Z".

I have built a multioutput classifier using logistic regression as the base classifier in scikit. It looks like each label classifier is an independent binary classifier. My question is, how can I compare the probabilities output by each classifier? As I said, I ultimately want to be able to compare the likelihood of a given label with that of the other labels in order to rank the certainty of their appearing. I know logistic regression outputs well-calibrated models, but are the probabilities of the 25 binary classifiers directly comparable? Would calibrating these classifiers help to ensure that their output probabilities are comparable?

Answer 1

I can't see any reason why they wouldn't be comparable, but I don't know what I could cite to convince a skeptic of this. They're probabilities. As long as P(L present) + P(L not present) == 1 for all labels L (which they have to be to be probabilities), the probabilities of different labels should be comparable.

Nevertheless, there may be cases where, for instance, labels A and B both have probability greater than any other label, but the two are mutually exclusive, so if label A is present then label B is guaranteed to be absent, and vice-versa. These kinds of correlations will not be reflected in their probabilities. But that's OK, because, well, multidimensional distributions have means and also covariance matrices, and those are different.