30
$\begingroup$

I have a classification task where I have a number of predictors (one of which is the most informative), and I am using the MARS model to construct my classifier (I am interested in any simple model, and using glms for illustrative purposes would be fine too). Now I have a huge class imbalance in the training data (about 2700 negative samples for each positive sample). Similar to Information Retrieval tasks, I am more concerned about predicting the top ranking positive test samples. For this reason, the performance on Precision Recall curves is important to me.

First of all, I simply trained the model on my training data keeping the class imbalance as it is. I visualize my trained model in red, and the most important input in blue.

Training on unbalanced data, evaluation on unbalanced data:

PR for unbalanced training ROC for unbalanced training

Thinking that the class imbalance is throwing the model off, since learning the top ranking positive samples is a miniscule part of the whole data set, I upsampled the positive training points to get a balanced training data set. When I plot the performance on the balanced training set, I get good performance. In both the PR and ROC curves, my trained model does better then the inputs.

Training on (upsampled) balanced data, evaluation also on (upsampled) balanced data:

PR for balanced training, visualised on balanced dataset ROC for balanced training, visualised on balanced dataset

However, if I use this model trained on the balanced data, to predict on the original, unbalanced training set, I still get bad performance on the PR curve.

Training on (upsampled) balanced data, evaluation on original unbalanced data:

PR for balanced training, visualised on original, unbalanced dataset ROC for balanced training, visualised on original, unbalanced dataset

So my questions are:

  1. Is the reason the visualization of the PR curve shows inferior performance of my trained model (red), while ROC curve shows improvements because of the class imbalance?
  2. Can resampling/up-sampling/down-sampling approaches resolve this to force the training to focus on the high precision/low recall region?
  3. Is there any other way to focus training on the high precision/low recall region?
$\endgroup$
  • $\begingroup$ Could you edit your question to clarify which measures are computed on the training set and which on held out data? $\endgroup$ – Jack Tanner Jan 24 '12 at 15:19
  • $\begingroup$ @JackTanner, everything is computed on the training set for now. Since the model does not have that many parameters, and the number of samples in the training set is huge I don't worry too much about overfitting. Besides, I want to be sure I am getting good performance on the training set before I can expect in in the test set. $\endgroup$ – highBandWidth Jan 24 '12 at 17:37
  • $\begingroup$ What knob are you controlling in your learning algorithm to evaluate precision at different recall levels? Have you tried to expand your feature set, e.g., with feature combinations and transformations? $\endgroup$ – Jack Tanner Jan 25 '12 at 6:16
  • $\begingroup$ @JackTanner, The model that I have (MARS with logit function) gives outputs in the range of 0 to 1, similar to logistic regression. It's basically the same, but includes a few more features. To get precision at different recalls, I simply set the thresholds at different points. I just use the standard way to calculate PR or ROC from a ranked list. $\endgroup$ – highBandWidth Jan 25 '12 at 20:48
15
$\begingroup$
  1. The ROC curve is insensitive to changes in class imbalance; see Fawcett (2004) "ROC Graphs: Notes and Practical Considerations for Researchers".
  2. Up-sampling the low-frequency class is a reasonable approach.
  3. There are many other ways of dealing with class imbalance. Boosting and bagging are two techniques that come to mind. This seems like a relevant recent study: Comparing Boosting and Bagging Techniques With Noisy and Imbalanced Data

P.S. Neat problem; I'd love to know how it turns out.

$\endgroup$
1
$\begingroup$

A recent study "An insight into classification with imbalanced data: Empirical results and current trends on using data intrinsic characteristics" compares three methods of improved classification on unbalanced data:

  • Data Sampling (as suggested in the question)
  • Algorithm modification
  • Cost sensitive learning
$\endgroup$
1
$\begingroup$

I wanted to draw attention to the fact, that the last 2 experiments are in fact using the SAME model on ALMOST THE SAME dataset. The difference in performance is not model difference, it is explained by different distributions of validation dataset and the properties of particular METRICS used - precision and recall, that depend highly on that distribution. To elaborate this point a bit more, if you took X distinct entries from your initial validation dataset and replicated the minority class for the upscaled dataset, your model will make the same predictions for those X entries, correct or incorrect, in both upscaled and unbalanced validation datasets. The only difference is that for each false positive there will be less true positives in the initial dataset (hence lower precision) and more true positives in the balanced dataset (simply due to the fact that there are more positive examples in the dataset in general). This is why Precision and Recall are said to be sensitive to skew. On the other hand, as your experiments illustrate as well, ROC does not change. This can be observed by looking at its definition as well. That's why ROC is said to not be sensitive to skew.

I don't yet have good answers for points 2 and 3 as am looking for those myself :)

$\endgroup$
0
$\begingroup$

Assuming the upsampled positive samples have the "same distribution" as in the "original set". As the number of positive samples increases, few changes happen

1) the number of TruePositives (TP) increases for "all thresholds" and, as a result, ratios TP/(TP+FP) and TP/(TP+FN) increase for all thresholds. So that the area under PRC is increasing.

2)the expected precision, also called precision of "dumb" model, increases from ~1/2700 (in original set) to ~1/2 (in case of "ideal" balance). Assuming your model performs better then the "dumb" model means that the area under curve will be more then 0.00037 in "original set" and more then 0.5 in the ideally balanced set.

3) while training the model on upscaled dataset, some models may "overfit" positive samples.

In regard to ROC curves, ROC curves are known to show little effect from class distribution variations (upscaling has very minor effect on FPR, while you can see some effect on TPR).

In regard to focusing in high precision/low recall region, you can optimize with respect to a cost function where False Positives are penalized more then False Negatives.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.