In a recent, well recieved, question, Tim asks when is unbalanced data really a problem in Machine Learning? The premise of the question is that there is a lot of machine learning literature discussing class balance and the problem of imbalanced classes. The idea is that datasets with an imbalance between the positive and negative class cause problems for some machine learning classification (I'm including probabilistic models here) algorithms, and methods should be sought to "balance" the dataset, restoring the perfect 50/50 split between positive and negative classes.

The general sense of the upvoted answers is that "it's not, at least if you are thoughtful in your modeling". M. Henry L., in an up-voted comment to an accepted answer, states

[...] there isn't a low level problem with using unbalanced data. In my experience, the advice to "avoid unbalanced data" is either algorithm-specific, or inherited wisdom. I agree with AdamO that in general, unbalanced data poses no conceptual problem to a well-specified model.

AdamO argues that the "problem" with class balance is really one of class rarity

Therefore, at least in regression (but I suspect in all circumstances), the only problem with imbalanced data is that you effectively have small sample size. If any method is suitable for the number of people in the rarer class, there should be no issue if their proportion membership is imbalanced.

If this is the true issue at hand, it leaves an open question: what is the purpose of all the resampling methods intended to balance the dataset: oversampling, undersampling, SMOTE, etc? Clearly they don't address the problem of implicitly having a small sample size, you can't create information out of nothing!

  • $\begingroup$ That's exactly what I'd have changed it to... thanks. It doesn't cover the entire scope of your question but a title doesn't have to-- it does clearly get at what kind of thing you're asking about. $\endgroup$ – Glen_b Jun 14 '17 at 1:58
  • $\begingroup$ There are certainly situations where bootstrap and subsampling methods that are useful and sometimes better than other nonparametric methods. Books on the bootstrap and subsampling cover this. There are discussions on this site that discuss this including superiority of the bootstrap over leave-one-out in discriminant analysis even in relatively small samples. There are certainly some situations where the bootstrap fails and those are mentioned in my book as well as other. $\endgroup$ – Michael Chernick Jun 14 '17 at 2:34
  • $\begingroup$ @MichaelChernick I'm not talking about the bootstrap, that's what Glen was commenting about. I'm speaking of "class balancing" approaches like over and under sampling so that the positive to negative class ase equally represented in a data set. $\endgroup$ – Matthew Drury Jun 14 '17 at 2:38
  • $\begingroup$ Do you include subsampling? Are you referring to unequal sample size only? How general a statement are you making? $\endgroup$ – Michael Chernick Jun 14 '17 at 2:56
  • $\begingroup$ @MichaelChernick I added some clarifying remarks in the first and last paragraphs, I hope that helps. $\endgroup$ – Matthew Drury Jun 14 '17 at 3:41

Some sampling techniques are to adjust for bias (if the population rate is known and different), but I agree with the notion that the unbalanced class is not the problem itself. One major reason comes down to processing performance. If our targeted class, for example, is an extreme rare case at 1:100000, our modeling dataset would be massive and computation would be difficult. Sampling, no matter what the strategy, is always throwing away some data in order to reduce the total dataset size. I suppose the difference among all the different sampling strategies, is just cleverness around which data do we throw away without sacrificing a loss in predictive possibilities.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.