I have a binary classification problem, where given a thing I need to determine whether it's of class A or class B. Now, I also have additional information: For each 30 examples for which I need to make that decision I know that at most 1 is of class A, other are of class B.
How can I use that info to improve my decision making?
I can of course simply pick a candidate with the highest percentage of being class A and say that the rest is B. But I'm also interested how this affects confidence scores. Is there an ML algorithm that accounts for this? I could divide each confidence by the sum of all confidence scores. Would that be a reasonable way to do it?
Edited to clarify that for each batch of 30 examples there is at most 1 example of class A. The amount of training examples is much larger.