83
$\begingroup$

We already had multiple questions about unbalanced data when using logistic regression, SVM, decision trees, bagging and a number of other similar questions, what makes it a very popular topic! Unfortunately, each of the questions seems to be algorithm-specific and I didn't find any general guidelines for dealing with unbalanced data.

Quoting one of the answers by Marc Claesen, dealing with unbalanced data

(...) heavily depends on the learning method. Most general purpose approaches have one (or several) ways to deal with this.

But when exactly should we worry about unbalanced data? Which algorithms are mostly affected by it and which are able to deal with it? Which algorithms would need us to balance the data? I am aware that discussing each of the algorithms would be impossible on a Q&A site like this. I am rather looking for general guidelines on when it could be a problem.

$\endgroup$
8
  • 3
    $\begingroup$ Possible duplicate of What is the root cause of the class imbalance problem? $\endgroup$ Jun 2 '17 at 14:04
  • $\begingroup$ @MatthewDrury thanks, this is an interesting question, but IMHO, it has a different scope. What I'm asking is for guidelines when this is really a problem. Surely answering the why question leads to answering the when question, but I'm looking for precise answer for the when question. $\endgroup$
    – Tim
    Jun 2 '17 at 14:12
  • 15
    $\begingroup$ Fair enough! I'm with you. The "literature" on this seems to be all about how to fix a problem, without bothering to convince you that there is in fact a problem to be solved, or even telling you in what situations a problem occurs or not. One of the most frustrating parts of the subject for me. $\endgroup$ Jun 2 '17 at 14:39
  • 4
    $\begingroup$ @MatthewDrury that is exactly the problem! $\endgroup$
    – Tim
    Jun 2 '17 at 14:43
  • 2
    $\begingroup$ A total survey of methods is not within the scope of an SE question. Do you want to refine the question? $\endgroup$
    – AdamO
    Jun 7 '17 at 16:02
42
+50
$\begingroup$

Not a direct answer, but it's worth noting that in the statistical literature, some of the prejudice against unbalanced data has historical roots.

Many classical models simplify neatly under the assumption of balanced data, especially for methods like ANOVA that are closely related to experimental design—a traditional / original motivation for developing statistical methods.

But the statistical / probabilistic arithmetic gets quite ugly, quite quickly, with unbalanced data. Prior to the widespread adoption of computers, the by-hand calculations were so extensive that estimating models on unbalanced data was practically impossible.

Of course, computers have basically rendered this a non-issue. Likewise, we can estimate models on massive datasets, solve high-dimensional optimization problems, and draw samples from analytically intractable joint probability distributions, all of which were functionally impossible like, fifty years ago.

It's an old problem, and academics sank a lot of time into working on the problem...meanwhile, many applied problems outpaced / obviated that research, but old habits die hard...

Edit to add:

I realize I didn't come out and just say it: there isn't a low level problem with using unbalanced data. In my experience, the advice to "avoid unbalanced data" is either algorithm-specific, or inherited wisdom. I agree with AdamO that in general, unbalanced data poses no conceptual problem to a well-specified model.

$\endgroup$
6
  • 6
    $\begingroup$ While I seem to get your point, your premises lack arguments backing them. Could you give some arguments and/or examples on the prejudice and on how did if affect machine learning? $\endgroup$
    – Tim
    Jun 7 '17 at 7:18
  • 1
    $\begingroup$ While what you say is mostly true, it is also the case that methods like anova is more robust with balanced data, nonnormality is less of an issue with balanced data, for example. But I believe all this is orthogonal to the intent of this question ... $\endgroup$ Jun 7 '17 at 15:47
  • 7
    $\begingroup$ I realize I didn't come out and just say it: there isn't a low level problem with using unbalanced data. In my experience, the advice to "avoid unbalanced data" is either algorithm-specific, or inherited wisdom. I agree with AdamO that in general, unbalanced data poses no conceptual problem to a well-specified model. $\endgroup$
    – Henry
    Jun 8 '17 at 4:59
  • 2
    $\begingroup$ @M.HenryL. this comment is worth adding to your answer for completeness. $\endgroup$
    – Tim
    Jun 13 '17 at 10:38
  • 1
    $\begingroup$ This doesn't seem to answer the question... the question is about the so-called class imbalance problem (i.e., lack of balance in a categorical dependent variable), whereas this answer is about lack of balance in predictors and/or grouping/clustering factors (as in ANOVA and mixed models). They are almost completely different issues $\endgroup$ Jun 27 '20 at 22:52
26
$\begingroup$

Unbalanced data is only a problem depending on your application. If for example your data indicates that A happens 99.99% of the time and 0.01% of the time B happens and you try to predict a certain result your algorithm will probably always say A. This is of course correct! It is unlikely for your method to get better prediction accuracy than 99.99%. However in many applications we are not interested in just the correctness of the prediction but also in why B happens sometimes. This is where unbalanced data becomes a problem. Because it is hard to convince your method that it can predict better than 99.99% correct. The method is correct but not for your question. So solving unbalanced data is basically intentionally biasing your data to get interesting results instead of accurate results. All methods are vulnerable although SVM and logistic regressions tend to be a little less vulnerable while decision trees are very vulnerable.

In general there are three cases:

  1. You are purely interested in accurate prediction and you think your data is representative. In this case you do not have to correct at all. Bask in the glory of your 99.99% accurate predictions :).

  2. You are interested in prediction but your data is from a fair sample but somehow you lost a number of observations. If you lost observations in a completely random way you're still fine. If you lost them in a biased way but you don't know how biased, you will need new data. However if these observations are lost only on the basis of one charateristic. (for example you sorted results in A and B but not in any other way but lost half of B) Ypu can bootstrap your data.

  3. You are not interested in accurate global prediction, but only in a rare case. In this case you can inflate the data of that case by bootstrapping the data or if you have enough data throwing a way data of the other cases. Notice that this does bias your data and results and so chances and that kind of results are wrong!

In general it mostly depends on what the goal is. Some goals suffer from unbalanced data others don't. All general prediction methods suffer from it because otherwise they would give terrible results in general.

$\endgroup$
12
  • 7
    $\begingroup$ How does this story change when we evalate our models probabilistically? $\endgroup$ Jun 6 '17 at 14:25
  • 1
    $\begingroup$ @AdamO you are ofcourse correct that logistic regression is still vunerable , I would just argue that it is less so. $\endgroup$
    – zen
    Jun 8 '17 at 8:16
  • 3
    $\begingroup$ @MatthewDrury Stephen Senn has an excellent discussion about this point in a paper I reread often. Heuristically, the odds ratio from a 2x2 table with entries a b c d is estimated by ad/(bc) and has variance 1/a+1/b+1/c+1/d. You can sample arbitrarily few cases (a and c) and the odds ratio is still unbiased, but the variance goes to infinity. It is a precision issue. $\endgroup$
    – AdamO
    Jun 8 '17 at 15:07
  • 2
    $\begingroup$ This presumes implicitly (1) that the KPI we attempt to maximize is accuracy, and (2) that accuracy is an appropriate KPI for classification model evaluation. It isn't. $\endgroup$ Jul 17 '18 at 5:58
  • 1
    $\begingroup$ @AdamO, isn't the bias spoken about wrt logistic regression in the coefficients/odds ratio etc., not the probability estimates which is presumably what is being discussed here wrt classifier performance $\endgroup$
    – seanv507
    Jan 22 '19 at 18:18
18
$\begingroup$

WLOG you can focus on imbalance in a single factor, rather than a more nuanced concept of "data sparsity", or small cell counts.

In statistical analyses not focused on learning, we are faced with the issue of providing adequate inference while controlling for one or more effects through adjustment, matching, or weighting. All of these have similar power and yield similar estimates to propensity score matching. Propensity score matching will balance the covariates in the analysis set. They all end up being "the same" in terms of reducing bias, maintaining efficiency because they block confounding effects. With imbalanced data, you may naively believe that your data are sufficiently large, but with a sparse number of people having the rarer condition: variance inflation diminishes power substantially, and it can be difficult to "control" for effects when those effects are strongly associated with the predictor and outcome.

Therefore, at least in regression (but I suspect in all circumstances), the only problem with imbalanced data is that you effectively have smaller sample size than the $N$ might represent. If any method is suitable for the number of people in the rarer class, there should be no issue if their proportion membership is imbalanced.

$\endgroup$
10
$\begingroup$

Let's assume we have two classes:

  • A, representing 99.99% of the population
  • B, representing 0.01% of the population

Let's assume we are interested in identifying class B elements, that could be individuals affected by a rare disease or fraudster.

Just by guessing A learners would score high on their loss-functions and the very few incorrectly classified elements might not move, numerically, the needle (in a haystack, in this case). This example brings the intuition behind one of the "tricks" to mitigate the class imbalance problem: tweaking the cost function.

I feel that unbalanced data is a problem when models show near-zero sensitivity and near-one specificity. See the example in this article under the section "ignoring the problem".

Problems have often a solution. Alongside the aforementioned trick, there are other options. However, they come at a price: an increase in model and computational complexity.

The question asks which models are more likely to settle on near-zero sensitivity and near-one specificity. I feel that it depends on a few dimensions:

  • Less capacity, as usual.
  • Some cost functions might struggle more than others: mean squared error (MSE) is less exposed than Huber - MSE should be less benign towards incorrectly classified B class elements.
$\endgroup$
5
  • 5
    $\begingroup$ This presumes implicitly (1) that the KPI we attempt to maximize is accuracy, and (2) that accuracy is an appropriate KPI for classification model evaluation. It isn't. $\endgroup$ Jul 17 '18 at 5:51
  • $\begingroup$ @StephanKolassa Looks like even if the KPI is not accuracy, say logistic regression's loss function is cross entropy, and given 100 negative vs 1 positive imbalanced data, it should still be likely that the model might not be able to tune the weights based on the 1 positive examples, hence the trained model might not give reasonable prediction for positive ones. $\endgroup$
    – avocado
    Mar 15 at 11:56
  • $\begingroup$ @avocado: both cross entropy and the more common log likelihood are proper scoring rules, so a logistic regression should indeed be able to find the correct predictions. Why do you think differently? Incidentally, why do you write "reasonable prediction for positive ones"? Why would a probabilistic classifier output "unreasonable" predictions for only positive classes? Yes, of course the predicted probabilities will be below one, but that reflects the ex post selection, i.e., a kind of bias. $\endgroup$ Mar 15 at 12:41
  • $\begingroup$ @StephanKolassa I think you're right that cross entropy should be good for logistic regression to learn and give correct predictions. But the un-answered question is why down-sampling negative examples is wildly used in industry? E.g. ctr prediction $\endgroup$
    – avocado
    Mar 15 at 13:23
  • 2
    $\begingroup$ @avocado: why over-/undersampling is popular is indeed a puzzling question. Some of the comments to my question here go into this; e.g., Sycorax' comment. $\endgroup$ Mar 15 at 13:28
6
$\begingroup$

If you think about it: On a perfectly separable highly imbalanced data set, almost any algorithm will perform without errors.

Hence, it is more a problem of noise in data and less tied to a particular algorithm. And you don't know beforehand which algorithm compensates for one particular type of noise best.

In the end you just have to try different methods and decide by cross validation.

$\endgroup$
3
  • 1
    $\begingroup$ I feel this comment is a bit under-appreciated. I just spend a bit of time convincing someone that class imbalance is not always a problem. $\endgroup$
    – RDK
    May 25 '18 at 20:51
  • 1
    $\begingroup$ This does not answer the question. How are unbalanced classes "more a problem of noise in data"? $\endgroup$ Jul 17 '18 at 5:46
  • 4
    $\begingroup$ @StephanKolassa It is an answer, because it says unbalanced data is not (directly) a problem. Hence you cannot ask "how" it is. For the more general question "how to deal with noise problems in data analysis", the answer is, that it is specific to individual data sets and all you can do is set up validation and try whatever works. If you really would like some discussion, I believe ele.uri.edu/faculty/he/PDFfiles/ImbalancedLearning.pdf has ideas. But in the end you would do sampling/reweighting/thresholding and it's not worth knowing what exactly happened in this data set. $\endgroup$
    – Gerenuk
    Jul 17 '18 at 12:02
5
$\begingroup$

I know I'm late to the party, but: the theory behind the data imbalance problem has been beautifully worked out by Sugiyama (2000) and a huge number of highly cited papers following that, under the keyword "covariate shift adaptation". There is also a whole book devoted to this subject by Sugiyama / Kawanabe from 2012, called "Machine Learning in Non-Stationary Environments". For some reason, this branch of research is only rarely mentioned in discussions about learning from imbalanced datasets, possibly because people are unaware of it?

The gist of it is this: data imbalance is a problem if a) your model is misspecified, and b) you're either interested in good performance on a minority class or you're interested in the model itself.

The reason can be illustrated very simply: if the model does not describe reality correctly, it will minimize the deviation from the most frequently observed type of samples (figure taken from Berk et al. (2018)): enter image description here

I will try to give a very brief summary of the technical main idea of Sugiyama. Suppose your training data are drawn from a distribution $p_{\mathrm{train}}(x)$, but you would like the model to perform well on data drawn from another distribution $p_{\mathrm{target}}(x)$. This is what's called "covariate shift", and it can also simply mean that you would like the model to work equally well on all regions of the data space, i.e. $p_{\mathrm{target}}(x)$ may be a uniform distribution. Then, instead of minimizing the expected loss over the training distribution

$$ \theta^* = \arg \min_\theta E[\ell(x, \theta)]_{p_{\text{train}}} \approx \arg \min_\theta \frac{1}{N}\sum_{i=1}^N \ell(x_i, \theta)$$

as one would usually do, one minimizes the expected loss over the target distribution:

$$ \theta^* = \arg \min_\theta E[\ell(x, \theta)]_{p_{\text{target}}} \\ = \arg \min_\theta E\left[\frac{p_{\text{target}}(x)}{p_{\text{train}}(x)}\ell(x, \theta)\right]_{p_{\text{train}}} \\ \approx \arg \min_\theta \frac{1}{N}\sum_{i=1}^N \underbrace{\frac{p_{\text{target}}(x_i)}{p_{\text{train}}(x_i)}}_{=w_i} \ell(x_i, \theta)$$

In practice, this amounts to simply weighting individual samples by their importance $w_i$. The key to practically implementing this is an efficient method for estimating the importance, which is generally nontrivial. This is one of the main topics of papers on this subject, and many methods can be found in the literature (keyword "Direct importance estimation").

All the oversampling / undersampling / SMOTE techniques people use are essentially just different hacks for implementing importance weighting, I believe.

$\endgroup$
6
  • $\begingroup$ >>>@"The reason can be illustrated very simply: if the model does not describe reality correctly, it will minimize the deviation from the most frequently observed type of samples" you mean the baseline estimator, but what when the real distribution is 90% to 10% in an binary classifier? $\endgroup$ Mar 24 at 13:15
  • $\begingroup$ @MaxKleiner I'm not sure I understand your question correctly, could you elaborate a bit? $\endgroup$
    – jhin
    Mar 31 at 16:20
  • $\begingroup$ One one side we have an actual distribution of 90 to 10. On the other side we downsample the unbalanced data (50 to 50) to prevent accuracy of the baseline estimator which has always an accuracy of 90%. Q: is the downsample (regularize unbalanced data) needed or not? $\endgroup$ Apr 6 at 15:25
  • 1
    $\begingroup$ @MaxKleiner You're touching on an important point there. The covariate shift literature often makes it sound as if ptarget were the actual real-world distribution you want to apply the estimator to. But ptarget should be tailored to what you really want, e.g., putting equal weight on different groups or regions of the data space. $\endgroup$
    – jhin
    May 6 at 8:09
  • 1
    $\begingroup$ Reweighting to handle a different target distribution is well understood.The main use case is reducing cost by undersampling the majority class for training data $\endgroup$
    – seanv507
    Jun 18 at 6:48
2
$\begingroup$

Great answers above, and not sure how much I can add here, but I feel there are three things to consider with imbalanced data, and new trade-offs you'll have to consider when rebalancing. I'd like to frame this in the context of predicting a minority outcome (a common task with imbalanced classes):

  1. By resampling, you may improve overall accuracy, but usually its the case that with severe class imbalance, you're actually trying to predict or otherwise describe features of the minority class. The best evaluation metrics here would then be F1 scores, precision, recall and the like. The resampling process (whether by SMOTE, or undersampling the majority class, etc.) disrupts the distribution of your data which naturally occurs, and training performed on these artificially created classes will usually perform poorly when applied to back to natural distributions. This creates sampling bias, in a sense. In my own work, I've found Random Forest Classifier does a bit better job than Logistic Regression, although it's pretty "data hungry" and will require enough minority samples (see point #2). It really depends on the question you're trying to answer and the types of data you have available.

  2. It's quite possible you may already have enough of the minority class to make useful predictions. Consider a class imbalance of 100:1. Would 1000 majority samples and 10 minority samples make for a useful classifier? Of course not. But what about 1,000,000 majority and 10,000 minority samples? The model you select may have enough of the minority outcome to make useful predictions. It's minority count - not relative proportion to the majority class - that is ultimately important.

  3. More of a general point is that we've become obsessed with correcting class imbalances, as if its a central problem. Left by the wayside are the far more important tasks of feature engineering and proper model selection as is necessary for predicting minority outcomes.

I wrote a piece about this here: Why Balancing Classes is Over-Hyped

$\endgroup$
2
  • $\begingroup$ Great article @Gabe Verzino. I found the pitfalls you mentioned to be spot on for the classification problem I was working on. My data has an imbalance of 4:1, and balancing the data affected the performance when the model was supplied with real-world data. I had a fair amount of data, 400k samples for the majority class and 100k for the minority class. For my use case, adding more data was better for generalization than balancing the data. $\endgroup$
    – Raqib
    Sep 22 at 18:06
  • $\begingroup$ That's awesome! Nice work. $\endgroup$ Sep 22 at 21:03
0
$\begingroup$

For me the most important problem with unbalanced data is the baseline estimator. For example, you have two classes with 90% and 10% sample distribution. But what does this mean for a dummy or naive classifier? You can infer this meaning by comparing it with a baseline’s performance. You can always predict the most frequent label in the training set, so the model has to be better or at least than 90% (this is the baseline)!

Typical baselines include those supported by scikit-learn’s "dummy" estimators:

Classification baselines:

  • "stratified": generates predictions by respecting the training set’s class distribution.
  • "most_frequent": always predicts the most frequent label in the training set.
  • "prior": always predicts the class that maximizes the class prior.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.