This question is different from Binary classification when one class consists of multiple subclasses
I have two classes that I want to distinguish by a supervised learning classifier such as a random forest, A vs. B. Furthermore, there are two kinds of As: A1 and A2. I've built a classifier A vs. B, a random forest, and just for fun also A1 vs. (A2+B) and A2 vs. (A1+B). The issue is, the AUC ROC is higher for both A1 vs. (A2+B) and A2 vs. (A1+B) than for the original one.
I would understand if it were the other way round: A1 and A2 could be random subsets of A. However, if it can easily detect A1 and A2, it stands to reason that (A1+A2) should also be easily distinguishable from B. Perhaps I expect too much from AUC ROC as an overall quality measure of a binary classifier.
Can someone point me to the relevant properties of AUC ROC or binary classifiers in general? Perhaps some classical book on machine learning sheds light on it?