44
$\begingroup$

We already had multiple questions about unbalanced data when using logistic regression, SVM, decision trees, bagging and a number of other similar questions, what makes it a very popular topic! Unfortunately, each of the questions seems to be algorithm-specific and I didn't find any general guidelines for dealing with unbalanced data.

Quoting one of the answers by Marc Claesen, dealing with unbalanced data

(...) heavily depends on the learning method. Most general purpose approaches have one (or several) ways to deal with this.

But when exactly should we worry about unbalanced data? Which algorithms are mostly affected by it and which are able to deal with it? Which algorithms would need us to balance the data? I am aware that discussing each of the algorithms would be impossible on Q&A site like this, I am rather looking for a general guidelines on when it could be a problem.

$\endgroup$
  • 3
    $\begingroup$ Possible duplicate of What is the root cause of the class imbalance problem? $\endgroup$ – Matthew Drury Jun 2 '17 at 14:04
  • $\begingroup$ @MatthewDrury thanks, this is an interesting question, but IMHO, it has a different scope. What I'm asking is for guidelines when this is really a problem. Surely answering the why question leads to answering the when question, but I'm looking for precise answer for the when question. $\endgroup$ – Tim Jun 2 '17 at 14:12
  • 8
    $\begingroup$ Fair enough! I'm with you. The "literature" on this seems to be all about how to fix a problem, without bothering to convince you that there is in fact a problem to be solved, or even telling you in what situations a problem occurs or not. One of the most frustrating parts of the subject for me. $\endgroup$ – Matthew Drury Jun 2 '17 at 14:39
  • 2
    $\begingroup$ @MatthewDrury that is exactly the problem! $\endgroup$ – Tim Jun 2 '17 at 14:43
  • 1
    $\begingroup$ A total survey of methods is not within the scope of an SE question. Do you want to refine the question? $\endgroup$ – AdamO Jun 7 '17 at 16:02
23
+50
$\begingroup$

Not a direct answer, but it's worth noting that in the statistical literature, some of the prejudice against unbalanced data has historical roots.

Many classical models simplify neatly under the assumption of balanced data, especially for methods like ANOVA that are closely related to experimental design—a traditional / original motivation for developing statistical methods.

But the statistical / probabilistic arithmetic gets quite ugly, quite quickly, with unbalanced data. Prior to the widespread adoption of computers, the by-hand calculations were so extensive that estimating models on unbalanced data was practically impossible.

Of course, computers have basically rendered this a non-issue. Likewise, we can estimate models on massive datasets, solve high-dimensional optimization problems, and draw samples from analytically intractable joint probability distributions, all of which were functionally impossible like, fifty years ago.

It's an old problem, and academics sank a lot of time into working on the problem...meanwhile, many applied problems outpaced / obviated that research, but old habits die hard...

Edit to add:

I realize I didn't come out and just say it: there isn't a low level problem with using unbalanced data. In my experience, the advice to "avoid unbalanced data" is either algorithm-specific, or inherited wisdom. I agree with AdamO that in general, unbalanced data poses no conceptual problem to a well-specified model.

$\endgroup$
  • 3
    $\begingroup$ While I seem to get your point, your premises lack arguments backing them. Could you give some arguments and/or examples on the prejudice and on how did if affect machine learning? $\endgroup$ – Tim Jun 7 '17 at 7:18
  • 1
    $\begingroup$ While what you say is mostly true, it is also the case that methods like anova is more robust with balanced data, nonnormality is less of an issue with balanced data, for example. But I believe all this is orthogonal to the intent of this question ... $\endgroup$ – kjetil b halvorsen Jun 7 '17 at 15:47
  • 6
    $\begingroup$ I realize I didn't come out and just say it: there isn't a low level problem with using unbalanced data. In my experience, the advice to "avoid unbalanced data" is either algorithm-specific, or inherited wisdom. I agree with AdamO that in general, unbalanced data poses no conceptual problem to a well-specified model. $\endgroup$ – Henry Jun 8 '17 at 4:59
  • 1
    $\begingroup$ @M.HenryL. this comment is worth adding to your answer for completeness. $\endgroup$ – Tim Jun 13 '17 at 10:38
15
$\begingroup$

WLOG you can focus on imbalance in a single factor, rather than a more nuanced concept of "data sparsity", or small cell counts.

In statistical analyses not focused on learning, we often find that similar or better power is afforded when using propensity scores to match the smaller group to the larger group. This is partly because matching serves a similar purpose to confounder adjustment in terms of "balancing" the determinants of group membership, thus blocking their confounding effects. The rationale for the number of confounders to possibly adjust for in a multivariate analysis depends on the sample size. Some rules of thumb say one variable per every 10 to 20 observations. In imbalanced data, you naively believe that your data are sufficiently large, but with a sparse number of people having the rarer condition: variance inflation diminishes power substantially. You are, in fact, over adjusting.

Therefore, at least in regression (but I suspect in all circumstances), the only problem with imbalanced data is that you effectively have small sample size. If any method is suitable for the number of people in the rarer class, there should be no issue if their proportion membership is imbalanced.

$\endgroup$
8
$\begingroup$

Unbalanced data is only a problem depending on your application. If for example your data indicates that A happens 99.99% of the time and 0.01% of the time B happens and you try to predict a certain result your algorithm will probably always say A. This is of course correct! It is unlikely for your method to get better prediction accuracy than 99.99%. However in many applications we are not interested in just the correctness of the prediction but also in why B happens sometimes. This is where unbalanced data becomes a problem. Because it is hard to convince your method that it can predict better than 99.99% correct. The method is correct but not for your question. So solving unbalanced data is basically intentionally biasing your data to get interesting results instead of accurate results. All methods are vulnerable although SVM and logistic regressions tend to be a little less vulnerable while decision trees are very vulnerable.

In general there are three cases:

  1. your purely interested in accurate prediction and you think your data is reprenstative. In this case you do not have to correct at all, Bask in the glory of your 99.99% accurate predictions :).

  2. You are interested in prediction but your data is from a fair sample but somehow you lost a number of observations. If you lost observations in a completely random way you're still fine. If you lost them in a biased way but you don't know how biased, you will need new data. However if these observations are lost only on the basis of one charateristic. (for example you sorted results in A and B but not in any other way but lost half of B) Ypu can bootstrap your data.

  3. You are not interested in accurate global prediction, but only in a rare case. In this case you can inflate the data of that case by bootstrapping the data or if you have enough data throwing a way data of the other cases. Notice that this does bias your data and results and so chances and that kind of results are wrong!

In general it mostly depends on what the goal is. Some goals suffer from unbalanced data others don't. All general prediction methods suffer from it because otherwise they would give terrible results in general.

$\endgroup$
  • 7
    $\begingroup$ How does this story change when we evalate our models probabilistically? $\endgroup$ – Matthew Drury Jun 6 '17 at 14:25
  • $\begingroup$ @MatthewDrury The probabalities from the original model are mostly correct for cases 1 and 3. The issue is that only with very large datasets B becomes correctly separable from A and the probablity of B slowly converges to its real value. The exception being that if B is very clearly separated from A or completely randomly separated from A, the probabalities will respectively almost immediately or never converge. $\endgroup$ – zen Jun 6 '17 at 14:36
  • $\begingroup$ @zen I rather disagree that logistic regression is less vulnerable. Logistic regression is quite vulnerable to data imbalance, it creates small sample bias and the log odds ratios tend toward a factor of 2. Conditional logistic regression is an alternative to estimating the same ORs without bias. $\endgroup$ – AdamO Jun 7 '17 at 16:11
  • $\begingroup$ @AdamO Yes logistic regression is still vulnerable. But for trees small cases can be completely ignored. it is not just small sample size either. even for large n and for example 80%-20% distribution between options trees can still opt for choosing the 80% option even if the fraction of the 20% option clearly increases with some variable x. If a new more extreme observation is found or if the number of branches is to low for any extreme point the tree will predict the 80% option while logistic regression will be less likely to do so. You are right about conditional logistical regression $\endgroup$ – zen Jun 7 '17 at 21:54
  • 1
    $\begingroup$ @MatthewDrury Stephen Senn has an excellent discussion about this point in a paper I reread often. Heuristically, the odds ratio from a 2x2 table with entries a b c d is estimated by ad/(bc) and has variance 1/a+1/b+1/c+1/d. You can sample arbitrarily few cases (a and c) and the odds ratio is still unbiased, but the variance goes to infinity. It is a precision issue. $\endgroup$ – AdamO Jun 8 '17 at 15:07
7
$\begingroup$

Let's assume we have two classes:

  • A, representing 99.99% of the population
  • B, representing 0.01% of the population

Let's assume we are interested in identifying class B elements, that could be individuals affected by a rare disease or fraudster.

Just by guessing A learners would score high on their loss-functions and the very few incorrectly classified elements might not move, numerically, the needle (in a haystack, in this case). This example brings the intuition behind one of the "tricks" to mitigate the class imbalance problem: tweaking the cost function.

I feel that unbalanced data is a problem when models show near-zero sensitivity and near-one specificity. See the example in this article under the section "ignoring the problem".

Problems have often a solution. Alongside the aforementioned trick, there are other options. However, they come at a price: an increase in model and computational complexity.

The question asks which models are more likely to settle on near-zero sensitivity and near-one specificity. I feel that it depends on a few dimensions:

  • Less capacity, as usual.
  • Some cost functions might struggle more than others: mean squared error (MSE) is less exposed than Huber - MSE should be less benign towards incorrectly classified B class elements.
$\endgroup$
  • 1
    $\begingroup$ This presumes implicitly (1) that the KPI we attempt to maximize is accuracy, and (2) that accuracy is an appropriate KPI for classification model evaluation. It isn't. $\endgroup$ – Stephan Kolassa Jul 17 '18 at 5:51
5
$\begingroup$

If you think about it: On a perfectly separable highly imbalanced data set, almost any algorithm will perform without errors.

Hence, it is more a problem of noise in data and less tied to a particular algorithm. And you don't know beforehand which algorithm compensates for one particular type of noise best.

In the end you just have to try different methods and decide by cross validation.

$\endgroup$
  • $\begingroup$ I feel this comment is a bit under-appreciated. I just spend a bit of time convincing someone that class imbalance is not always a problem. $\endgroup$ – RDK May 25 '18 at 20:51
  • $\begingroup$ This does not answer the question. How are unbalanced classes "more a problem of noise in data"? $\endgroup$ – Stephan Kolassa Jul 17 '18 at 5:46
  • 2
    $\begingroup$ @StephanKolassa It is an answer, because it says unbalanced data is not (directly) a problem. Hence you cannot ask "how" it is. For the more general question "how to deal with noise problems in data analysis", the answer is, that it is specific to individual data sets and all you can do is set up validation and try whatever works. If you really would like some discussion, I believe ele.uri.edu/faculty/he/PDFfiles/ImbalancedLearning.pdf has ideas. But in the end you would do sampling/reweighting/thresholding and it's not worth knowing what exactly happened in this data set. $\endgroup$ – Gerenuk Jul 17 '18 at 12:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.