Does up-sampling lead to lots of false positives in production?

Question

Say we have a dataset with a binary outcome variable that takes the positive case (outcome = 1) roughly 20% of the time. Often, we would modify the training set by down-sampling the 0's such that the training set has something like a 50/50 split in the outcomes between 0's and 1's.

When this model is in production, though, wouldn't it experience a lot of false positives? Its perception of the "baseline" or "average" (e.g. intercept in logistic regression) seems to be too high. It seems to me like it would end up predicting too many 1's. Am I off base here?

Answer 1

Yes, this is based on my answer for a very similar question from the Data Science SE

Yes, the classifier will expect the relative class frequencies in operation to be the same as those in the training set. This means that if you over-sample the minority class in the training set, the classifier is likely to over-predict that class in operational use.

To see why it is best to consider probabilistic classifiers, where the decision is based on the posterior probability of class membership p(C_i|x), but this can be written using Bayes' rule as

$p(C_i|x) = \frac{p(x|C_i)p(C_i)}{p(x)}\qquad$ where $\qquad p(x) = \sum_j p(x|C_j)p(c_j)$,

so we can see that the decision depends on the prior probabilities of the classes, $p(C_i)$, so if the prior probabilities in the training set are different than those in operation, the operational performance of our classifier will be sub-optimal, even if it is optimal for the training set conditions.

Some classifiers have a problem learning from imbalanced datasets, so one solution is to oversample the classes by just the right amount to ameliorate this bias in the classifier. However, this is really difficult, because this tends to be a problem only when the data are very scarce, which unfortunately means you don't have enough data to estimate the amount of correction required. Fortunately this bias tends to go away rapidly as the size of the dataset increases, so if you have lots of data, you don't need to do anything.

If you have a balanced training set you can post-process the output to compensate for the difference in training set and operational priors. We take the output of the classifier trained on an oversampled dataset and multiply by the ratio of operational and training set prior probabilities,

$q_o(C_i|x) \propto p_t(x|C_i)p_t(C_i) \times \frac{p_o(C_i)}{p_t(C_i} = p_t(x|C_i)p_o(C_i)$

Quantities with the o subscript relate to operational conditions and those wit the t subscript relate to training set conditions. I have written this as $q_o(C_i|x)$ as it is an un-normalised probability, but it is straight forward to renormalise them by dividing by the sum of $q_o(C_i|x)$ over all classes.

If you are using logistic regression, then resampling the data probably has no benefits over just changing the threshold (which should depend on the false-positive and false-negative misclassification costs, and the degree of imbalance is entirely irrelevant).

TLDR; Resampling is not necessary if you have lots of data, unless you are using a classifier that doesn't support cost-sensitive learning, or if the dataset is very small. If you are going to resample, then the amount of resampling should depend on the false-positive and false-negative costs, rather than the degree of imbalance.

Answer 2

I will answer this question in a bit of a rephrased form: "What is the best sampling scheme with imbalanced data?"

If you take a look at my answer below, I link to a few different articles and explain how to deal with imbalanced data to create a stable classifier (whether that's in a live/production environment or in a static model. Obviously, live/production models are sensitive to data, so it's hard to really assess what will go on at first, but a good sampling scheme and loss function, will help address issues with imbalanced data.

The classic imbalanced sampling technique is SMOTE (see ref below), which oversamples from the minority class to synthetically increase its prevalence. Boosting algorithms (like adaboost)also will oversample the cases it got wrong, in order to fix issues with predictions. Focal is similar in that it will down-weight the "easy" predictors (in the loss function), so it makes sense to use it. The tricky part is that boosting algorithms are essentially prone to overfitting since their sampling is gradient-based to reduce error, so one must be always careful with how to introduce sampling schemes and loss functions. That's the only caveat with them. Below I've included all 3 references.

SMOTE: Chawla, Nitesh V., Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. "SMOTE: synthetic minority over-sampling technique." Journal of artificial intelligence research 16 (2002): 321-357.

Adaboost: Rätsch, Gunnar, Takashi Onoda, and K-R. Müller. "Soft margins for AdaBoost." Machine learning 42, no. 3 (2001): 287-320.

Focal: Lin, T. Y., Goyal, P., Girshick, R., He, K., & Dollár, P. (2017). Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision (pp. 2980-2988).

Does it make sense to use Focal loss for a tree based classifier like XGBoost?

The one big issue to consider (which was addressed in the above comment) is that for optimal performance, you also want to train the model on similar data it will see in production. You will hurt it by forcing a 50%-50% class balance if in reality, the live data will be 20-80. So, that's where things like a Focal loss, that focus on the loss function, or adaboost, which oversamples the "wrong" cases in each subsequent boosting tree to correct the errors. Oversampling the "minority class" might hurt.