How to oversample only the training set when using cross validation in sklearn

Question

I have an imbalanced dataset, and using oversampling to make it balanced. I am working on a regression problem. For testing purposes I am applying cross validation on the same dataset for now, using scikit-learn's LassoCV().

However, if I do oversampling before CV, then my model easily overfits since the oversampled items will be distributed over training and testing sets during cross validation. I wonder if there is a way to apply oversampling only on the training set after cross validation makes the split. I prefer to use a scikit-learn method, e.g., cross_val_predict. I wonder if it is possible? Or, do I need to write my own code for this task?

Answer 1

I would argue that you should perform no oversampling at all.

Oversampling almost always arises from using accuracy as a performance metric, since class imbalance can mean that your model with an impressive-looking $98\%$ accuracy is outperformed by a model that predicts the majority class every time and scores $99\%$ accuracy. This is related to my discussions here and here about comparing classification accuracy to the accuracy of such a baseline, "must-beat" model, and I believe that discussion to expose accuracy as being less useful that it might first seem.

However, accuracy is problematic even when the classes are balanced, and metrics like $F_1$ score, sensitivity, and specificity offer minimal improvement. This is because most models do not make hard classifications. They make predictions on a continuum (typically on the interval $[0,1]$), and the hard classifications come from applying some kind of threshold. This is troubling for multiple reasons.

1. The typical software default of $0.5$ might be wildly inappropriate (even for balanced classes, as Stephan Kolassa explains in his explanation about why accuracy is a problematic performance metric). If there is a small probability of something happening that would lead to a disaster if you miss it, you might be inclined to proceed as if that event will happen, even if it is unlikely. If this sounds silly, perhaps an example will help. There is (hopefully) a pretty small probability of being in a car accident, but there is so little cost to buckling when there is no accident and such a potential cost to not buckling when there is a wreck that most people buckle their seat belts.

2. It might be that multiple decisions can be made, despite there being only two categories. For instance, an extreme prediction either way might be taken as predicting a categorical outcome, but a prediction in the middle might be more of a signal to collect more information.

Hopefully, some of the below links can convince you of the probabilities being useful and that you need not mess with the data by oversampling in order to get accurate predicted probabilities.

Cross Validated: Why is accuracy not the best measure for assessing classification models?

Cross Validated: Are unbalanced datasets problematic, and (how) does oversampling (purport to) help?

Cross Validated: Academic reference on the drawbacks of accuracy, F1 score, sensitivity and/or specificity

Cross Validated: Calculating the Brier or log score from the confusion matrix, or from accuracy, sensitivity, specificity, F1 score etc

Cross Validated: Upweight minority class vs. downsample+upweight majority class?

Cross Validated Meta: Profusion of threads on imbalanced data - can we merge/deem canonical any?

Frank Harrell's Blog: Classification vs. Prediction

Frank Harrell's Blog: Damage Caused by Classification Accuracy and Other Discontinuous Improper Accuracy Scoring Rules

These links are (shamelessly) taken from an answer I posted in the past few days. I bring this up because class imbalance and what models output seem to be poorly understood by many machine learning practitioners.