4
$\begingroup$

I have an imbalanced dataset (10000 positives and 300 negatives) and have divided this into train and test sets. I perform oversampling/undersampling only on the train set since doing this on the test set would not represent a real-world scenario.

A Random Forest Classifier is able to classify the training set well (F-score of 0.92 for both positive and negative class) but performs badly on the test set (F-score of 0.83 for the positive class and 0.13 for the negative class).

Why does the classifier perform poorly on the test set although it has learnt to identify the difference between the two classes in the train set? Could it be because the distribution of the train set is now different from the test set? If so, how do I take care of this?

I came across this post but the answers are not particularly helpful.

$\endgroup$
9
  • $\begingroup$ Over/undersampling doesn't add any new information, it only replicates data, which is done to prevent the model from being biased, but still doesn't help the model to learn better. $\endgroup$ Sep 25, 2019 at 13:37
  • $\begingroup$ Yes, this was my understanding too. But why is the model performing so poorly on the test set even though it has learnt features from the train set? Also, if I perform oversampling/undersampling on the test set as well, I get good results on both the positive and negative classes. This led me to believe that sampling changes the distribution of the data. $\endgroup$
    – Anuj
    Sep 25, 2019 at 13:46
  • $\begingroup$ stats.stackexchange.com/questions/283170/… $\endgroup$
    – mkt
    Sep 25, 2019 at 13:55
  • $\begingroup$ Why do you think 0.83 is poor? It's normal for performance on the test set to be worse than on the training set $\endgroup$
    – mkt
    Sep 25, 2019 at 13:55
  • 1
    $\begingroup$ @mkt I appreciate the help! Thanks a lot! $\endgroup$
    – Anuj
    Sep 25, 2019 at 14:53

3 Answers 3

7
$\begingroup$

The answer to the title question is "of course it does"; you are shifting the distribution toward the minority class.

You can shift your model's predictions back to match the original distribution, see e.g. Convert predicted probabilities after downsampling to actual probabilities in classification or, equivalently, adjust the prediction threshold.

There's also a serious question on whether you needed to resample in the first place. See What is the root cause of the class imbalance problem?, When is unbalanced data really a problem in Machine Learning? If you do get better performance after balancing, with correct use of prediction thresholds/shifting, I'd like to know about it. I haven't been able to find a definitive answer on whether balancing helps a classifier learn. (Henry's answer to the second linked question here suggests not, but...)

$\endgroup$
2
  • $\begingroup$ Thanks for the reply! The links were very useful. Would the model perform better if the model's predictions are changed to match the original distribution i.e. would the formula mentioned in www3.nd.edu/~rjohns15/content/papers/ssci2015_calibrating.pdf actually improve the number of correct classifications? The authors of that paper have mentioned that the formula does not affect the ranking. $\endgroup$
    – Anuj
    Sep 25, 2019 at 18:22
  • $\begingroup$ The shift is a linear shift to log-odds, so it's monotonic on probabilities, so it doesn't affect rank-ordering (and hence neither the AUROC/AUPRC). It will affect the F1-score (and all confusion matrix based scores), but the same effect can be made by adjusting the threshold for class predictions. $\endgroup$ Sep 25, 2019 at 18:40
1
$\begingroup$

F1 score is the harmonic mean of precision and recall. Precision is defined as $$\text{precision} = P(Y = 1 | \hat Y = 1)$$ while recall is defined as $$ \text{recall} = P(\hat Y = 1 | Y = 1)$$ Recall does not care about the distribution of true class labels since it is being conditioned upon in the definition. Say you trained your model and the recall is 90%. Putting more positive samples in the test set will not change that. This is not true for precision. You could in principle throw in tons of negative samples in your test set, and the precision will be pushed towards zero. In short, your model might be doing ok. Comparing class distribution sensitive metrics to detect over/under fitting when you up/down sample training set will give you misleading result.

To detect over/under fitting in your case, I would suggest look at the ROC curve first. ROC is not sensitive to class distribution. If ROC AUC differs a lot between training and test set, then your are likely to over/under fit. Then you can look at things like average precision or others on test set.

$\endgroup$
1
$\begingroup$

I'd suggest avoiding oversampling and undersampling approaches. For a good classifier and performance metric it should not matter and it is unscientific. The F1 score is bias towards the majority class and is class distribution sensitive, see https://arxiv.org/ftp/arxiv/papers/1503/1503.06410.pdf. Also consider the ROC and ROC-AUC as an objective way of assessing performance which is suited for imbalanced data. In general it is better to look at a curve than rely on a single default cut-off like p=0.5 as the F1 score. I prefer the precision recall gain curve to precision recall, because it is standardised to the baseline. For PR curves it matters a lot which is defined as the positive class and is a deficiency of the approach.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.