AUC ROC when one class consists of smaller subclasses

Question

This question is different from Binary classification when one class consists of multiple subclasses

I have two classes that I want to distinguish by a supervised learning classifier such as a random forest, A vs. B. Furthermore, there are two kinds of As: A1 and A2. I've built a classifier A vs. B, a random forest, and just for fun also A1 vs. (A2+B) and A2 vs. (A1+B). The issue is, the AUC ROC is higher for both A1 vs. (A2+B) and A2 vs. (A1+B) than for the original one.

I would understand if it were the other way round: A1 and A2 could be random subsets of A. However, if it can easily detect A1 and A2, it stands to reason that (A1+A2) should also be easily distinguishable from B. Perhaps I expect too much from AUC ROC as an overall quality measure of a binary classifier.

Can someone point me to the relevant properties of AUC ROC or binary classifiers in general? Perhaps some classical book on machine learning sheds light on it?

Answer 1

AUC ROC is a measure which tells one how much a classifier separates those classes. One way to think of that is to consider for example predictive probability of an instance and to sort you observations ordered by the predicted probability to belong to one class. For a good discriminant you should see that in general that most instances of the first class have high predicted probabilities to belong to that class and low probability if belongs to the other class.

Now we will take an fabricated example. Let's suppose that your features are height and average daily calories consumed and you want to predict if the subjects have university degree (A) versus non university degree (B). I suppose that this would be a hard problem, since university degree have low correlation with the input features. In fact I would be surprised if one would get a significant discriminant far from random classification. Now let's consider that we separate those with university degrees in two categories: male university degree (A1) and female university degree (A2). Because we involve sex I suppose height would be a very good discriminant for sex. Usually males are taller than females, so height acts as a good discriminator between sexes.

What happens with ROC AUC in this case? If you predict A1,A2 vs B you have a low AUC because your features are not a good discriminant. If you predict A1 vs A2,B or A2 vs A1,B than height produces good classifiers which dominates the difference between any A subcategory and B. As a consequence AUC ROC will be higher.

As you can see what you noticed is that sometimes (I would say many times if you have a lot of features) you can isolate well some subgroups but the learning problem you have to solve remains hard.