Does oversampling/undersampling change the distribution of the data?

Question

I have an imbalanced dataset (10000 positives and 300 negatives) and have divided this into train and test sets. I perform oversampling/undersampling only on the train set since doing this on the test set would not represent a real-world scenario.

A Random Forest Classifier is able to classify the training set well (F-score of 0.92 for both positive and negative class) but performs badly on the test set (F-score of 0.83 for the positive class and 0.13 for the negative class).

Why does the classifier perform poorly on the test set although it has learnt to identify the difference between the two classes in the train set? Could it be because the distribution of the train set is now different from the test set? If so, how do I take care of this?

I came across this post but the answers are not particularly helpful.

Answer 1

The answer to the title question is "of course it does"; you are shifting the distribution toward the minority class.

You can shift your model's predictions back to match the original distribution, see e.g. Convert predicted probabilities after downsampling to actual probabilities in classification or, equivalently, adjust the prediction threshold.

There's also a serious question on whether you needed to resample in the first place. See What is the root cause of the class imbalance problem?, When is unbalanced data really a problem in Machine Learning? If you do get better performance after balancing, with correct use of prediction thresholds/shifting, I'd like to know about it. I haven't been able to find a definitive answer on whether balancing helps a classifier learn. (Henry's answer to the second linked question here suggests not, but...)

Answer 2

F1 score is the harmonic mean of precision and recall. Precision is defined as $$\text{precision} = P(Y = 1 | \hat Y = 1)$$ while recall is defined as $$ \text{recall} = P(\hat Y = 1 | Y = 1)$$ Recall does not care about the distribution of true class labels since it is being conditioned upon in the definition. Say you trained your model and the recall is 90%. Putting more positive samples in the test set will not change that. This is not true for precision. You could in principle throw in tons of negative samples in your test set, and the precision will be pushed towards zero. In short, your model might be doing ok. Comparing class distribution sensitive metrics to detect over/under fitting when you up/down sample training set will give you misleading result.

To detect over/under fitting in your case, I would suggest look at the ROC curve first. ROC is not sensitive to class distribution. If ROC AUC differs a lot between training and test set, then your are likely to over/under fit. Then you can look at things like average precision or others on test set.

Answer 3

I'd suggest avoiding oversampling and undersampling approaches. For a good classifier and performance metric it should not matter and it is unscientific. The F1 score is bias towards the majority class and is class distribution sensitive, see https://arxiv.org/ftp/arxiv/papers/1503/1503.06410.pdf. Also consider the ROC and ROC-AUC as an objective way of assessing performance which is suited for imbalanced data. In general it is better to look at a curve than rely on a single default cut-off like p=0.5 as the F1 score. I prefer the precision recall gain curve to precision recall, because it is standardised to the baseline. For PR curves it matters a lot which is defined as the positive class and is a deficiency of the approach.