What problem does oversampling, undersampling, and SMOTE solve?

Question

In a recent, well recieved, question, Tim asks when is unbalanced data really a problem in Machine Learning? The premise of the question is that there is a lot of machine learning literature discussing class balance and the problem of imbalanced classes. The idea is that datasets with an imbalance between the positive and negative class cause problems for some machine learning classification (I'm including probabilistic models here) algorithms, and methods should be sought to "balance" the dataset, restoring the perfect 50/50 split between positive and negative classes.

The general sense of the upvoted answers is that "it's not, at least if you are thoughtful in your modeling". M. Henry L., in an up-voted comment to an accepted answer, states

[...] there isn't a low level problem with using unbalanced data. In my experience, the advice to "avoid unbalanced data" is either algorithm-specific, or inherited wisdom. I agree with AdamO that in general, unbalanced data poses no conceptual problem to a well-specified model.

AdamO argues that the "problem" with class balance is really one of class rarity

Therefore, at least in regression (but I suspect in all circumstances), the only problem with imbalanced data is that you effectively have small sample size. If any method is suitable for the number of people in the rarer class, there should be no issue if their proportion membership is imbalanced.

If this is the true issue at hand, it leaves an open question: what is the purpose of all the resampling methods intended to balance the dataset: oversampling, undersampling, SMOTE, etc? Clearly they don't address the problem of implicitly having a small sample size, you can't create information out of nothing!

Answer 1

The problem that these methods are trying to solve is to increase the impact of minority class on cost function. This is because algos trying to fit well the whole dataset and then adapt to majority. Other approach would be to use class weights, and this aporoach in most cases gives better results, since there is no information loss by undersampling or performance loss and introduction of noise by oversampling.

Answer 2

Some sampling techniques are to adjust for bias (if the population rate is known and different), but I agree with the notion that the unbalanced class is not the problem itself. One major reason comes down to processing performance. If our targeted class, for example, is an extreme rare case at 1:100000, our modeling dataset would be massive and computation would be difficult. Sampling, no matter what the strategy, is always throwing away some data in order to reduce the total dataset size. I suppose the difference among all the different sampling strategies, is just cleverness around which data do we throw away without sacrificing a loss in predictive possibilities.

Answer 3

I will give you a more extreme example. Consider the case where you have a dataset with 99 data points labeled as positive and only one labeled as negative. During training, your model will realize that if it classifies everything as positive, it will end up getting away with it. One way of fixing this is to oversample the underrepresented class and another is to undersample the overrepresented class. For example, in a dataset of 70 positive and 30 negative labels, I might sample the negative labels with replacement and positive ones without replacement which will result in my model encountering more negative labels during training. This way, if my model tries to classify everything as positive, it will incurr greater loss than it would have otherwise.

One more approach that does not pertain to sampling is to adjust the cost function to give higher weights to the data points with the minority label. For example, if you are using NLL loss in a dataset where 1's are overrepresented compared to 0's among labels, you could adjust your loss function to be:

$L(\tilde{x_i}, y_i) = -\alpha(y_i)\ln(\tilde{x_i}) - \beta(1 - y_i) \ln(1 - \tilde{x_i})$

where $\beta > \alpha$. The magnitude of the diference $\beta - \alpha$ depends on the extent of overrepresentation/underrepresentation.

Answer 4

There are many technqiues for oversampling and undersampling to overcome the sparsity of minority in imbalanced data anv vice versa.... Yet most of them have consequence on behavior of your model (roughly speaking variance). I personally use an self-devised technique where oversampling and undersampling are done simoultanously. Spicing this Combined Sampling with Adapative Synthetic Sampling (ADASYN), I call it C-ADASYN. You may think of hair-transplant where there you oversample in sparse area and unersample in dense area, to keep the behaviour fair and add synthetic sample if needed to augment the population.

Answer 5

I'm going to disagree with the premise that unbalanced data isn't a problem in machine learning. Perhaps less so in regression, but it certainly is in classification.

Imbalanced Data is relevant in Machine Learning applications because of decreased performance of algorithms (the research I am thinking of is specifically on classifiers) in the setting of class imbalance.

Take a simple binary classification problem with 25:1 ratio of training examples of class A' vs. 'class B'. Research has shown that accuracy pertaining to the classification of class B takes a hit simply because of the decreased ratio of training data. Makes sense, as the less # of training examples you have, the poorer your classifier will train on that data. As one of the commenters stated, you can't make something out of nothing. From the papers I've seen, in multiclass classification problems, it seems you need to get to a 10:1 ratio to start having a significant impact on accuracy of the minority class. Perhaps folks who read different literature than I've seen have different opinions.

So, the proposed solutions are: Oversampling the minority class, Undersampling the majority class, or using SMOTE on the minority class. Yes, you can't really create data out of nowhere (SMOTE sort-of does, but not exactly) unless you're getting into synthetic data creation for the minority class (no simple method). Other techniques like MixUp and the like potentially fall into this concept, but I think that they are more regularizers than class imbalance solutions. In the papers I have read, Oversampling > SMOTE > Undersampling.

Regardless of your technique, you are altering the relationship between majority and minority classes which may affect incidence. In other words, if you are creating a classifier to detect super-rare brain disease X which has an incidence of 1 in 100,000 and your classifier is at 1:1, you might be more sensitive and less specific with a larger number of false positives. If it is important that you detect those cases and arbiter later, you're ok. If not, you wasted a lot of other people's time and money. This problem eventually will need to be dealt with.

So to answer the question:

tl/dr: Class-balancing operations like Over/Undersampling and SMOTE (and synthetic data) exist to improve machine learning algorithm (classifier) performance by resolving the inherent performance hit in an algorithm caused by the imbalance itself.