When is unbalanced data really a problem in Machine Learning?

Question

We already had multiple questions about unbalanced data when using logistic regression, SVM, decision trees, bagging and a number of other similar questions, what makes it a very popular topic! Unfortunately, each of the questions seems to be algorithm-specific and I didn't find any general guidelines for dealing with unbalanced data.

Quoting one of the answers by Marc Claesen, dealing with unbalanced data

(...) heavily depends on the learning method. Most general purpose approaches have one (or several) ways to deal with this.

But when exactly should we worry about unbalanced data? Which algorithms are mostly affected by it and which are able to deal with it? Which algorithms would need us to balance the data? I am aware that discussing each of the algorithms would be impossible on a Q&A site like this. I am rather looking for general guidelines on when it could be a problem.

Answer 1

Not a direct answer, but it's worth noting that in the statistical literature, some of the prejudice against unbalanced data has historical roots.

Many classical models simplify neatly under the assumption of balanced data, especially for methods like ANOVA that are closely related to experimental design—a traditional / original motivation for developing statistical methods.

But the statistical / probabilistic arithmetic gets quite ugly, quite quickly, with unbalanced data. Prior to the widespread adoption of computers, the by-hand calculations were so extensive that estimating models on unbalanced data was practically impossible.

Of course, computers have basically rendered this a non-issue. Likewise, we can estimate models on massive datasets, solve high-dimensional optimization problems, and draw samples from analytically intractable joint probability distributions, all of which were functionally impossible like, fifty years ago.

It's an old problem, and academics sank a lot of time into working on the problem...meanwhile, many applied problems outpaced / obviated that research, but old habits die hard...

Edit to add:

I realize I didn't come out and just say it: there isn't a low level problem with using unbalanced data. In my experience, the advice to "avoid unbalanced data" is either algorithm-specific, or inherited wisdom. I agree with AdamO that in general, unbalanced data poses no conceptual problem to a well-specified model.

Answer 2

Unbalanced data is only a problem depending on your application. If for example your data indicates that A happens 99.99% of the time and 0.01% of the time B happens and you try to predict a certain result your algorithm will probably always say A. This is of course correct! It is unlikely for your method to get better prediction accuracy than 99.99%. However in many applications we are not interested in just the correctness of the prediction but also in why B happens sometimes. This is where unbalanced data becomes a problem. Because it is hard to convince your method that it can predict better than 99.99% correct. The method is correct but not for your question. So solving unbalanced data is basically intentionally biasing your data to get interesting results instead of accurate results. All methods are vulnerable although SVM and logistic regressions tend to be a little less vulnerable while decision trees are very vulnerable.

In general there are three cases:

1. You are purely interested in accurate prediction and you think your data is representative. In this case you do not have to correct at all. Bask in the glory of your 99.99% accurate predictions :).

2. You are interested in prediction but your data is from a fair sample but somehow you lost a number of observations. If you lost observations in a completely random way you're still fine. If you lost them in a biased way but you don't know how biased, you will need new data. However if these observations are lost only on the basis of one charateristic. (for example you sorted results in A and B but not in any other way but lost half of B) Ypu can bootstrap your data.

3. You are not interested in accurate global prediction, but only in a rare case. In this case you can inflate the data of that case by bootstrapping the data or if you have enough data throwing a way data of the other cases. Notice that this does bias your data and results and so chances and that kind of results are wrong!

In general it mostly depends on what the goal is. Some goals suffer from unbalanced data others don't. All general prediction methods suffer from it because otherwise they would give terrible results in general.

Answer 3

WLOG you can focus on imbalance in a single factor, rather than a more nuanced concept of "data sparsity", or small cell counts.

In statistical analyses not focused on learning, we are faced with the issue of providing adequate inference while controlling for one or more effects through adjustment, matching, or weighting. All of these have similar power and yield similar estimates to propensity score matching. Propensity score matching will balance the covariates in the analysis set. They all end up being "the same" in terms of reducing bias, maintaining efficiency because they block confounding effects. With imbalanced data, you may naively believe that your data are sufficiently large, but with a sparse number of people having the rarer condition: variance inflation diminishes power substantially, and it can be difficult to "control" for effects when those effects are strongly associated with the predictor and outcome.

Therefore, at least in regression (but I suspect in all circumstances), the only problem with imbalanced data is that you effectively have smaller sample size than the $N$ might represent. If any method is suitable for the number of people in the rarer class, there should be no issue if their proportion membership is imbalanced.

Answer 4

I know I'm late to the party, but: the theory behind the data imbalance problem has been beautifully worked out by Sugiyama (2000) and a huge number of highly cited papers following that, under the keyword "covariate shift adaptation". There is also a whole book devoted to this subject by Sugiyama / Kawanabe from 2012, called "Machine Learning in Non-Stationary Environments". For some reason, this branch of research is only rarely mentioned in discussions about learning from imbalanced datasets, possibly because people are unaware of it?

The gist of it is this: data imbalance is a problem if a) your model is misspecified, and b) you're either interested in good performance on a minority class or you're interested in the model itself.

The reason can be illustrated very simply: if the model does not describe reality correctly, it will minimize the deviation from the most frequently observed type of samples (figure taken from Berk et al. (2018)):

I will try to give a very brief summary of the technical main idea of Sugiyama. Suppose your training data are drawn from a distribution $p_{\mathrm{train}}(x)$, but you would like the model to perform well on data drawn from another distribution $p_{\mathrm{target}}(x)$. This is what's called "covariate shift", and it can also simply mean that you would like the model to work equally well on all regions of the data space, i.e. $p_{\mathrm{target}}(x)$ may be a uniform distribution. Then, instead of minimizing the expected loss over the training distribution

$$ \theta^* = \arg \min_\theta E[\ell(x, \theta)]_{p_{\text{train}}} \approx \arg \min_\theta \frac{1}{N}\sum_{i=1}^N \ell(x_i, \theta)$$

as one would usually do, one minimizes the expected loss over the target distribution:

$$ \theta^* = \arg \min_\theta E[\ell(x, \theta)]_{p_{\text{target}}} \\ = \arg \min_\theta E\left[\frac{p_{\text{target}}(x)}{p_{\text{train}}(x)}\ell(x, \theta)\right]_{p_{\text{train}}} \\ \approx \arg \min_\theta \frac{1}{N}\sum_{i=1}^N \underbrace{\frac{p_{\text{target}}(x_i)}{p_{\text{train}}(x_i)}}_{=w_i} \ell(x_i, \theta)$$

In practice, this amounts to simply weighting individual samples by their importance $w_i$. The key to practically implementing this is an efficient method for estimating the importance, which is generally nontrivial. This is one of the main topics of papers on this subject, and many methods can be found in the literature (keyword "Direct importance estimation").

All the oversampling / undersampling / SMOTE techniques people use are essentially just different hacks for implementing importance weighting, I believe.

Answer 5

Let's assume we have two classes:

• A, representing 99.99% of the population
• B, representing 0.01% of the population

Let's assume we are interested in identifying class B elements, that could be individuals affected by a rare disease or fraudster.

Just by guessing A learners would score high on their loss-functions and the very few incorrectly classified elements might not move, numerically, the needle (in a haystack, in this case). This example brings the intuition behind one of the "tricks" to mitigate the class imbalance problem: tweaking the cost function.

I feel that unbalanced data is a problem when models show near-zero sensitivity and near-one specificity. See the example in this article under the section "ignoring the problem".

Problems have often a solution. Alongside the aforementioned trick, there are other options. However, they come at a price: an increase in model and computational complexity.

The question asks which models are more likely to settle on near-zero sensitivity and near-one specificity. I feel that it depends on a few dimensions:

• Less capacity, as usual.
• Some cost functions might struggle more than others: mean squared error (MSE) is less exposed than Huber - MSE should be less benign towards incorrectly classified B class elements.

Answer 6

If you think about it: On a perfectly separable highly imbalanced data set, almost any algorithm will perform without errors.

Hence, it is more a problem of noise in data and less tied to a particular algorithm. And you don't know beforehand which algorithm compensates for one particular type of noise best.

In the end you just have to try different methods and decide by cross validation.

Answer 7

Great answers above, and not sure how much I can add here, but I feel there are three things to consider with imbalanced data, and new trade-offs you'll have to consider when rebalancing. I'd like to frame this in the context of predicting a minority outcome (a common task with imbalanced classes):

1. By resampling, you may improve overall accuracy, but usually its the case that with severe class imbalance, you're actually trying to predict or otherwise describe features of the minority class. The best evaluation metrics here would then be F1 scores, precision, recall and the like. The resampling process (whether by SMOTE, or undersampling the majority class, etc.) disrupts the distribution of your data which naturally occurs, and training performed on these artificially created classes will usually perform poorly when applied to back to natural distributions. This creates sampling bias, in a sense. In my own work, I've found Random Forest Classifier does a bit better job than Logistic Regression, although it's pretty "data hungry" and will require enough minority samples (see point #2). It really depends on the question you're trying to answer and the types of data you have available.

2. It's quite possible you may already have enough of the minority class to make useful predictions. Consider a class imbalance of 100:1. Would 1000 majority samples and 10 minority samples make for a useful classifier? Of course not. But what about 1,000,000 majority and 10,000 minority samples? The model you select may have enough of the minority outcome to make useful predictions. It's minority count - not relative proportion to the majority class - that is ultimately important.

3. More of a general point is that we've become obsessed with correcting class imbalances, as if its a central problem. Left by the wayside are the far more important tasks of feature engineering and proper model selection as is necessary for predicting minority outcomes.

I wrote a piece about this here: Why Balancing Classes is Over-Hyped

Answer 8

For me the most important problem with unbalanced data is the baseline estimator. For example, you have two classes with 90% and 10% sample distribution. But what does this mean for a dummy or naive classifier? You can infer this meaning by comparing it with a baseline’s performance. You can always predict the most frequent label in the training set, so the model has to be better or at least than 90% (this is the baseline)!

Typical baselines include those supported by scikit-learn’s "dummy" estimators:

Classification baselines:

• "stratified": generates predictions by respecting the training set’s class distribution.
• "most_frequent": always predicts the most frequent label in the training set.
• "prior": always predicts the class that maximizes the class prior.