# Binary classification where I know only one candidate can be positive

I have a binary classification problem, where given a thing I need to determine whether it's of class A or class B. Now, I also have additional information: For each 30 examples for which I need to make that decision I know that at most 1 is of class A, other are of class B.
How can I use that info to improve my decision making?

I can of course simply pick a candidate with the highest percentage of being class A and say that the rest is B. But I'm also interested how this affects confidence scores. Is there an ML algorithm that accounts for this? I could divide each confidence by the sum of all confidence scores. Would that be a reasonable way to do it?

EDIT

Edited to clarify that for each batch of 30 examples there is at most 1 example of class A. The amount of training examples is much larger.

• I think you will not really succeed in reducing this problem to a binary classification problem. The reason is that class A is so heavily underrepresented that the algorithm will probably have a very hard time learning anything about class A at all... Maybe it would be better to understand this as an outlier detection problem (because I suspect that there is something very special about that A object), i.e. you could use isolation forests... Jan 15 '20 at 9:34
• Do you mean 30 out-of-sample values or 30 training values?
– Dave
Jan 15 '20 at 10:42
• "out-of-sample values" I guess is a good way to call it. I have lots of training data, so that's not a problem. Training data at the moment is just records with A or B, but I can group the records between which I know only one example is A. To explain in a different way, if I have 3000 training data points, a 100 of them is of class A. Jan 15 '20 at 10:58
• This is just conjecture, but I don't think you can use that information to improve performance, because it doesn't really tell you anything additional about the individual examples, which is where the classification occurs. Given the rarity of the class, you might consider downsampling examples of class B, but ~3% is not crazy rare. Jan 15 '20 at 11:28
• That changes the situation. Depending on how many predictors you have, you may have adequate data (though not if you want to estimate 3000 parameters). I haven’t decided what I think of this, but what about some kind of cross validation with out-of-sample sets with the same 29:1 imbalance ratio to tune the cutoff probability? Then when you’ve found the cutoff probability, train on all training data and use that model plus the cutoff probability to make your out-of-sample classifications. (This may result in two or zero in the group that’s supposed to have exactly one. Always expect errors.)
– Dave
Jan 15 '20 at 11:30