When is oversampling preferable to undersampling and vice versa?

Question

When data is unbalanced, that is, when the distribution of classes being predicted is very uneven (e.g. 90%/10% for two classes or 10%/15%/75% for three classes), many machine learning models have difficulty in correctly predicting the classes because they tend to overpredict that unlabeled data will result in the largest class. Probably the two most common approaches to helping machine learning models to make better predictions is oversampling (randomly duplicating examples of the underrepresented classes until they match the proportion of the largest class) or undersampling (randomly removing examples of the larger classes until they match the proportion of the smallest class).

I have read various questions related to my interest here, but I have not found one that asks the same thing:

• What problem does oversampling, undersampling, and SMOTE solve?
• When is unbalanced data really a problem in Machine Learning?

My question is this: under which conditions is oversampling preferable to undersampling, and under which conditions is undersampling preferable to undersampling?

As a corollary question, I realize that some algorithms like SMOTE have some advantages that I don't fully understand. If an answer could explain the superiority of alternate approaches (under which conditions) to both over- and undersampling, I would appreciate that also.

I understand that the over- or undersampling technique should be done as part of the training of the models, not before any split. So, when I ask which is "preferable", I mean from the perspective of achieving highest accuracy in the test data. I do realize that it is possible that one or other of over- or undersampling might be preferable depending on if the priority is to predict overall accuracy versus highest accuracy of the minority class; I hope that answers might clarify such nuances.

Answer 1

Much of the apparent issue with class imbalance seems to come from using an improper accuracy performance metric that can give an impressive-looking $98\%$ when $99\%$ of the observations belong to one class, meaning that predicting that one class every time would yield higher accuracy. Consequently, the best move is probably to leave the data as they are and use better measures of performance, such as log-loss (negative binomial log likelihood) and Brier score (mean squared error).

If you must fiddle with the data, an advantage that oversampling has over undersampling is that oversampling does not discard data.

A scenario where undersampling can make sense is when it comes to data collection, which is the topic of the King and Zeng (2001) paper mentioned here. (There is a ton of good information in that link.) However, that paper assumes that you have yet to go through the trouble of collecting data. Once you have done so, I see minimal reason to discard data.

Another scenario where is might make sense to undersample is if you cannot fit the entire data set in memory. This would be true after the data collection, though the better remedy for this might be to go acquire better hardware.