Optimising for Precision-Recall curves under class imbalance

Question

I have a classification task where I have a number of predictors (one of which is the most informative), and I am using the MARS model to construct my classifier (I am interested in any simple model, and using glms for illustrative purposes would be fine too). Now I have a huge class imbalance in the training data (about 2700 negative samples for each positive sample). Similar to Information Retrieval tasks, I am more concerned about predicting the top ranking positive test samples. For this reason, the performance on Precision Recall curves is important to me.

First of all, I simply trained the model on my training data keeping the class imbalance as it is. I visualize my trained model in red, and the most important input in blue.

Training on unbalanced data, evaluation on unbalanced data:

Thinking that the class imbalance is throwing the model off, since learning the top ranking positive samples is a miniscule part of the whole data set, I upsampled the positive training points to get a balanced training data set. When I plot the performance on the balanced training set, I get good performance. In both the PR and ROC curves, my trained model does better then the inputs.

Training on (upsampled) balanced data, evaluation also on (upsampled) balanced data:

However, if I use this model trained on the balanced data, to predict on the original, unbalanced training set, I still get bad performance on the PR curve.

Training on (upsampled) balanced data, evaluation on original unbalanced data:

So my questions are:

1. Is the reason the visualization of the PR curve shows inferior performance of my trained model (red), while ROC curve shows improvements because of the class imbalance?
2. Can resampling/up-sampling/down-sampling approaches resolve this to force the training to focus on the high precision/low recall region?
3. Is there any other way to focus training on the high precision/low recall region?

Answer 1

1. The ROC curve is insensitive to changes in class imbalance; see Fawcett (2004) "ROC Graphs: Notes and Practical Considerations for Researchers".
2. Up-sampling the low-frequency class is a reasonable approach.
3. There are many other ways of dealing with class imbalance. Boosting and bagging are two techniques that come to mind. This seems like a relevant recent study: Comparing Boosting and Bagging Techniques With Noisy and Imbalanced Data

P.S. Neat problem; I'd love to know how it turns out.

Answer 2

A recent study "An insight into classification with imbalanced data: Empirical results and current trends on using data intrinsic characteristics" compares three methods of improved classification on unbalanced data:

• Data Sampling (as suggested in the question)
• Algorithm modification
• Cost sensitive learning

Answer 3

I wanted to draw attention to the fact, that the last 2 experiments are in fact using the SAME model on ALMOST THE SAME dataset. The difference in performance is not model difference, it is explained by different distributions of validation dataset and the properties of particular METRICS used - precision and recall, that depend highly on that distribution. To elaborate this point a bit more, if you took X distinct entries from your initial validation dataset and replicated the minority class for the upscaled dataset, your model will make the same predictions for those X entries, correct or incorrect, in both upscaled and unbalanced validation datasets. The only difference is that for each false positive there will be less true positives in the initial dataset (hence lower precision) and more true positives in the balanced dataset (simply due to the fact that there are more positive examples in the dataset in general). This is why Precision and Recall are said to be sensitive to skew. On the other hand, as your experiments illustrate as well, ROC does not change. This can be observed by looking at its definition as well. That's why ROC is said to not be sensitive to skew.

I don't yet have good answers for points 2 and 3 as am looking for those myself :)

Answer 4

Assuming the upsampled positive samples have the "same distribution" as in the "original set". As the number of positive samples increases, few changes happen

1) the number of TruePositives (TP) increases for "all thresholds" and, as a result, ratios TP/(TP+FP) and TP/(TP+FN) increase for all thresholds. So that the area under PRC is increasing.

2)the expected precision, also called precision of "dumb" model, increases from ~1/2700 (in original set) to ~1/2 (in case of "ideal" balance). Assuming your model performs better then the "dumb" model means that the area under curve will be more then 0.00037 in "original set" and more then 0.5 in the ideally balanced set.

3) while training the model on upscaled dataset, some models may "overfit" positive samples.

In regard to ROC curves, ROC curves are known to show little effect from class distribution variations (upscaling has very minor effect on FPR, while you can see some effect on TPR).

In regard to focusing in high precision/low recall region, you can optimize with respect to a cost function where False Positives are penalized more then False Negatives.