I have a multi class and multi label problem: each sample can be labelled with a number of labels between 1 and n out of N. So I train N binary classifiers, so that each of those can say if the example belongs or not to this class. Each of those binary classifiers gives for each sample the probability of the sample being in that class.
In practice, to classify a new sample I evaluate this with all N binary classifiers, getting a probability score for each label, and then assign the top n labels, as far as its probability is higher than a threshold.
For example, suppose there are the classes A, B, C, D, E, F, and each sample x belongs to up to 3 classes.
I can get the following matrix:
labels A B C D E F p(x) .8 .5 .3 .2 .2 .1
then I should predict this as A, B, since are the higher probabilities, and ignore C because it's lower than 0.5.
So far, so good, this makes sense for me. However, those 5 binary classifiers are not all equally good at their job. Perhaps classifier A is really bad, and classifier B is not. So I tried to model those different performances for each classifier as prior probability, then I take the precision of each model as the probability of P(x=A) given that A binary model says its A.
With this fix, the example tuns into this:
labels A B C D E F precision(model) .5 .9 .8 .8 .8 .8 p(x) .8 .5 .3 .2 .2 .1 p*precision .4 .45 .24 .16 .16 .08
Then I should apply the threshold to the p*precision.
Is this approach correct? should I use an extra operation to soft the probabilities? (to avoid labels always been selected just because its binary model is too good, like .99).
For the binary classification task, I'm using naive bayes from sklearn.naive_bayes.MultinomialNB, and precision is calculated for each model on its test set.