# When is unbalanced data really a problem in Machine Learning?

We already had multiple questions about unbalanced data when using logistic regression, SVM, decision trees, bagging and a number of other similar questions, what makes it a very popular topic! Unfortunately, each of the questions seems to be algorithm-specific and I didn't find any general guidelines for dealing with unbalanced data.

Quoting one of the answers by Marc Claesen, dealing with unbalanced data

(...) heavily depends on the learning method. Most general purpose approaches have one (or several) ways to deal with this.

But when exactly should we worry about unbalanced data? Which algorithms are mostly affected by it and which are able to deal with it? Which algorithms would need us to balance the data? I am aware that discussing each of the algorithms would be impossible on Q&A site like this, I am rather looking for a general guidelines on when it could be a problem.

• Possible duplicate of What is the root cause of the class imbalance problem? – Matthew Drury Jun 2 '17 at 14:04
• @MatthewDrury thanks, this is an interesting question, but IMHO, it has a different scope. What I'm asking is for guidelines when this is really a problem. Surely answering the why question leads to answering the when question, but I'm looking for precise answer for the when question. – Tim Jun 2 '17 at 14:12
• Fair enough! I'm with you. The "literature" on this seems to be all about how to fix a problem, without bothering to convince you that there is in fact a problem to be solved, or even telling you in what situations a problem occurs or not. One of the most frustrating parts of the subject for me. – Matthew Drury Jun 2 '17 at 14:39
• @MatthewDrury that is exactly the problem! – Tim Jun 2 '17 at 14:43
• A total survey of methods is not within the scope of an SE question. Do you want to refine the question? – AdamO Jun 7 '17 at 16:02

## 6 Answers

Not a direct answer, but it's worth noting that in the statistical literature, some of the prejudice against unbalanced data has historical roots.

Many classical models simplify neatly under the assumption of balanced data, especially for methods like ANOVA that are closely related to experimental design—a traditional / original motivation for developing statistical methods.

But the statistical / probabilistic arithmetic gets quite ugly, quite quickly, with unbalanced data. Prior to the widespread adoption of computers, the by-hand calculations were so extensive that estimating models on unbalanced data was practically impossible.

Of course, computers have basically rendered this a non-issue. Likewise, we can estimate models on massive datasets, solve high-dimensional optimization problems, and draw samples from analytically intractable joint probability distributions, all of which were functionally impossible like, fifty years ago.

It's an old problem, and academics sank a lot of time into working on the problem...meanwhile, many applied problems outpaced / obviated that research, but old habits die hard...

Edit to add:

I realize I didn't come out and just say it: there isn't a low level problem with using unbalanced data. In my experience, the advice to "avoid unbalanced data" is either algorithm-specific, or inherited wisdom. I agree with AdamO that in general, unbalanced data poses no conceptual problem to a well-specified model.

• While I seem to get your point, your premises lack arguments backing them. Could you give some arguments and/or examples on the prejudice and on how did if affect machine learning? – Tim Jun 7 '17 at 7:18
• While what you say is mostly true, it is also the case that methods like anova is more robust with balanced data, nonnormality is less of an issue with balanced data, for example. But I believe all this is orthogonal to the intent of this question ... – kjetil b halvorsen Jun 7 '17 at 15:47
• I realize I didn't come out and just say it: there isn't a low level problem with using unbalanced data. In my experience, the advice to "avoid unbalanced data" is either algorithm-specific, or inherited wisdom. I agree with AdamO that in general, unbalanced data poses no conceptual problem to a well-specified model. – Henry Jun 8 '17 at 4:59
• @M.HenryL. this comment is worth adding to your answer for completeness. – Tim Jun 13 '17 at 10:38
• This doesn't seem to answer the question... the question is about the so-called class imbalance problem (i.e., lack of balance in a categorical dependent variable), whereas this answer is about lack of balance in predictors and/or grouping/clustering factors (as in ANOVA and mixed models). They are almost completely different issues – Jake Westfall Jun 27 '20 at 22:52

Unbalanced data is only a problem depending on your application. If for example your data indicates that A happens 99.99% of the time and 0.01% of the time B happens and you try to predict a certain result your algorithm will probably always say A. This is of course correct! It is unlikely for your method to get better prediction accuracy than 99.99%. However in many applications we are not interested in just the correctness of the prediction but also in why B happens sometimes. This is where unbalanced data becomes a problem. Because it is hard to convince your method that it can predict better than 99.99% correct. The method is correct but not for your question. So solving unbalanced data is basically intentionally biasing your data to get interesting results instead of accurate results. All methods are vulnerable although SVM and logistic regressions tend to be a little less vulnerable while decision trees are very vulnerable.

In general there are three cases:

1. your purely interested in accurate prediction and you think your data is reprenstative. In this case you do not have to correct at all, Bask in the glory of your 99.99% accurate predictions :).

2. You are interested in prediction but your data is from a fair sample but somehow you lost a number of observations. If you lost observations in a completely random way you're still fine. If you lost them in a biased way but you don't know how biased, you will need new data. However if these observations are lost only on the basis of one charateristic. (for example you sorted results in A and B but not in any other way but lost half of B) Ypu can bootstrap your data.

3. You are not interested in accurate global prediction, but only in a rare case. In this case you can inflate the data of that case by bootstrapping the data or if you have enough data throwing a way data of the other cases. Notice that this does bias your data and results and so chances and that kind of results are wrong!

In general it mostly depends on what the goal is. Some goals suffer from unbalanced data others don't. All general prediction methods suffer from it because otherwise they would give terrible results in general.

• How does this story change when we evalate our models probabilistically? – Matthew Drury Jun 6 '17 at 14:25
• @MatthewDrury The probabalities from the original model are mostly correct for cases 1 and 3. The issue is that only with very large datasets B becomes correctly separable from A and the probablity of B slowly converges to its real value. The exception being that if B is very clearly separated from A or completely randomly separated from A, the probabalities will respectively almost immediately or never converge. – zen Jun 6 '17 at 14:36
• @zen I rather disagree that logistic regression is less vulnerable. Logistic regression is quite vulnerable to data imbalance, it creates small sample bias and the log odds ratios tend toward a factor of 2. Conditional logistic regression is an alternative to estimating the same ORs without bias. – AdamO Jun 7 '17 at 16:11
• @MatthewDrury Stephen Senn has an excellent discussion about this point in a paper I reread often. Heuristically, the odds ratio from a 2x2 table with entries a b c d is estimated by ad/(bc) and has variance 1/a+1/b+1/c+1/d. You can sample arbitrarily few cases (a and c) and the odds ratio is still unbiased, but the variance goes to infinity. It is a precision issue. – AdamO Jun 8 '17 at 15:07
• This presumes implicitly (1) that the KPI we attempt to maximize is accuracy, and (2) that accuracy is an appropriate KPI for classification model evaluation. It isn't. – Stephan Kolassa Jul 17 '18 at 5:58

WLOG you can focus on imbalance in a single factor, rather than a more nuanced concept of "data sparsity", or small cell counts.

In statistical analyses not focused on learning, we are faced with the issue of providing adequate inference while controlling for one or more effects through adjustment, matching, or weighting. All of these have similar power and yield similar estimates to propensity score matching. Propensity score matching will balance the covariates in the analysis set. They all end up being "the same" in terms of reducing bias, maintaining efficiency because they block confounding effects. With imbalanced data, you may naively believe that your data are sufficiently large, but with a sparse number of people having the rarer condition: variance inflation diminishes power substantially, and it can be difficult to "control" for effects when those effects are strongly associated with the predictor and outcome.

Therefore, at least in regression (but I suspect in all circumstances), the only problem with imbalanced data is that you effectively have smaller sample size than the $$N$$ might represent. If any method is suitable for the number of people in the rarer class, there should be no issue if their proportion membership is imbalanced.

Let's assume we have two classes:

• A, representing 99.99% of the population
• B, representing 0.01% of the population

Let's assume we are interested in identifying class B elements, that could be individuals affected by a rare disease or fraudster.

Just by guessing A learners would score high on their loss-functions and the very few incorrectly classified elements might not move, numerically, the needle (in a haystack, in this case). This example brings the intuition behind one of the "tricks" to mitigate the class imbalance problem: tweaking the cost function.

I feel that unbalanced data is a problem when models show near-zero sensitivity and near-one specificity. See the example in this article under the section "ignoring the problem".

Problems have often a solution. Alongside the aforementioned trick, there are other options. However, they come at a price: an increase in model and computational complexity.

The question asks which models are more likely to settle on near-zero sensitivity and near-one specificity. I feel that it depends on a few dimensions:

• Less capacity, as usual.
• Some cost functions might struggle more than others: mean squared error (MSE) is less exposed than Huber - MSE should be less benign towards incorrectly classified B class elements.
• This presumes implicitly (1) that the KPI we attempt to maximize is accuracy, and (2) that accuracy is an appropriate KPI for classification model evaluation. It isn't. – Stephan Kolassa Jul 17 '18 at 5:51

If you think about it: On a perfectly separable highly imbalanced data set, almost any algorithm will perform without errors.

Hence, it is more a problem of noise in data and less tied to a particular algorithm. And you don't know beforehand which algorithm compensates for one particular type of noise best.

In the end you just have to try different methods and decide by cross validation.

• I feel this comment is a bit under-appreciated. I just spend a bit of time convincing someone that class imbalance is not always a problem. – RDK May 25 '18 at 20:51
• This does not answer the question. How are unbalanced classes "more a problem of noise in data"? – Stephan Kolassa Jul 17 '18 at 5:46
• @StephanKolassa It is an answer, because it says unbalanced data is not (directly) a problem. Hence you cannot ask "how" it is. For the more general question "how to deal with noise problems in data analysis", the answer is, that it is specific to individual data sets and all you can do is set up validation and try whatever works. If you really would like some discussion, I believe ele.uri.edu/faculty/he/PDFfiles/ImbalancedLearning.pdf has ideas. But in the end you would do sampling/reweighting/thresholding and it's not worth knowing what exactly happened in this data set. – Gerenuk Jul 17 '18 at 12:02

I know I'm late to the party, but: the theory behind the data imbalance problem has been beautifully worked out by Sugiyama (2000) and a huge number of highly cited papers following that, under the keyword "covariate shift adaptation". There is also a whole book devoted to this subject by Sugiyama / Kawanabe from 2012, called "Machine Learning in Non-Stationary Environments". For some reason, this branch of research is only rarely mentioned in discussions about learning from imbalanced datasets, possibly because people are unaware of it?

The gist of it is this: data imbalance is a problem if a) your model is misspecified, and b) you're either interested in good performance on a minority class or you're interested in the model itself.

The reason can be illustrated very simply: if the model does not describe reality correctly, it will minimize the deviation from the most frequently observed type of samples (figure taken from Berk et al. (2018)):

I will try to give a very brief summary of the technical main idea of Sugiyama. Suppose your training data are drawn from a distribution $$p_{\mathrm{train}}(x)$$, but you would like the model to perform well on data drawn from another distribution $$p_{\mathrm{target}}(x)$$. This is what's called "covariate shift", and it can also simply mean that you would like the model to work equally well on all regions of the data space, i.e. $$p_{\mathrm{target}}(x)$$ may be a uniform distribution. Then, instead of minimizing the expected loss over the training distribution

$$\theta^* = \arg \min_\theta E[\ell(x, \theta)]_{p_{\text{train}}} \approx \arg \min_\theta \frac{1}{N}\sum_{i=1}^N \ell(x_i, \theta)$$

as one would usually do, one minimizes the expected loss over the target distribution:

$$\theta^* = \arg \min_\theta E[\ell(x, \theta)]_{p_{\text{target}}} \\ = \arg \min_\theta E\left[\frac{p_{\text{target}}(x)}{p_{\text{train}}(x)}\ell(x, \theta)\right]_{p_{\text{train}}} \\ \approx \arg \min_\theta \frac{1}{N}\sum_{i=1}^N \underbrace{\frac{p_{\text{target}}(x_i)}{p_{\text{train}}(x_i)}}_{=w_i} \ell(x_i, \theta)$$

In practice, this amounts to simply weighting individual samples by their importance $$w_i$$. The key to practically implementing this is an efficient method for estimating the importance, which is generally nontrivial. This is one of the main topics of papers on this subject, and many methods can be found in the literature (keyword "Direct importance estimation").

All the oversampling / undersampling / SMOTE techniques people use are essentially just different hacks for implementing importance weighting, I believe.