I had an online course, where I learned, that unbalanced classes in the training data might lead to problems, because classification algorithms go for the majority rule, as it gives good results if the unbalance is too much. In an assignment one had to balance the data via undersampling the majority class.

In this blog however, someone claims that balanced data is even worse:

https://matloff.wordpress.com/2015/09/29/unbalanced-data-is-a-problem-no-balanced-data-is-worse/

So which one is it? Should I balance the data or not? Does it depend on the algorithm used, as some might be able to adept to the unbalanced proportions of classes? If so, which ones are reliable on unbalanced data?

up vote 10 down vote accepted

The intuitive reasoning has been explained in the blogpost:

If our goal is Prediction, this will cause a definite bias. And worse, it will be a permanent bias, in the sense that we will not have consistent estimates as the sample size grows.

So, arguably the problem of (artificially) balanced data is worse than the unbalanced case.

Balanced data are good for classification, but you obviously loose information about appearance frequencies, which is going to affect accuracy metrics themselves, as well as production performance.

Let's say you're recognizing hand-written letters from English alphabet (26 letters). Overbalancing every letter appearance will give every letter a probability of being classified (correctly or not) roughly 1/26, so classifier will forget about actual distribution of letters in the original sample. And it's ok when classifier is able to generalize and recognize every letter with high accuracy.

But if accuracy and most importantly generalization isn't "so high" (I can't give you a definition - you can think of it just as a "worst case") - the misclassified points will most-likely equally distribute among all letters, something like:

"A" was misclassified 10 times
"B" was misclassified 10 times
"C" was misclassified 11 times
"D" was misclassified 10 times
...and so on

Without balancing (assuming that "A" and "C" have much higher probabilities of appearance in text)

"A" was misclassified 3 times
"B" was misclassified 14 times
"C" was misclassified 3 times
"D" was misclassified 14 times
...and so on

So frequent cases will get fewer misclassifications. Whether it's good or not depends on your task. For natural text recognition, one could argue that letters with higher frequencies are more viable, as they would preserve semantics of the original text, bringing the recognition task closer to prediction (where semantics represent tendencies). But if you're trying to recognize something like screenshot of ECDSA-key (more entropy -> less prediction) - keeping data unbalanced wouldn't help. So, again, it depends.

The most important distinction is that accuracy estimate is getting biased itself (as you can see in the balanced alphabet example), so you don't know how the model's behavior is getting affected by most rare or most frequent points.

P.S. You can always track performance of unbalanced classification with Precision/Recall metrics first and decide wether you need to add balancing or not.


EDIT: There is additional confusion that lies in estimation theory precisely in the difference between sample mean and population mean. For instance, you might know (arguably) actual distribution of English letters in the alphabet $p(x_i | \theta)$, but your sample (training set) is not large enough to estimate it correctly (with $p(x_i | \hat \theta)$). So in order to compensate for a $\hat \theta_i - \theta_i$, it is sometimes recommended to rebalance classes according to either population itself or parameters known from a larger sample (thus better estimator). However, in practice there is no guarantee that "larger sample" is identically distributed due to risk of getting biased data on every step (let's say English letters collected from technical literature vs fiction vs the whole library) so balancing could still be harmful.

This answer should also clarify applicability criteria for balancing:

The class imbalance problem is caused by there not being enough patterns belonging to the minority class, not by the ratio of positive and negative patterns itself per se. Generally if you have enough data, the "class imbalance problem" doesn't arise

As a conclusion, artificial balancing is rarely useful if training set is large enough. Absence of statistical data from a larger identically distributed sample also suggests no need for artificial balancing (especially for prediction), otherwise the quality of estimator is as good as "probability to meet a dinosaur":

What is the probability to meet a dinosaur out in the street?

1/2 you either meet a dinosaur or you do not meet a dinosaur

  • 3
    I think besides the explanation of the issue, the important take-away from this answer is that one should try unbalanced first and check its results and only if necessary do the balancing and check its result. +1 – Zelphir Mar 28 '17 at 11:35

Consistent with @kjetil-b-halvorsen's comment, the rapid adoption of machine learning has confused researchers about prediction vs. classification. As I described in more detail here, classification is only appropriate in a minority of cases. When the outcome is rare (or too common), probabilities are everything because in that case one can only reasonably speak about tendencies, not about predicting individual occurrences.

In statistics, we learned a while back that any method that requires one to exclude some of the data is highly suspect. So the goal of balancing outcomes is misplaced. Prediction of tendencies (probabilities) does not require it. And once you estimate a probability you can make an optimal decision by applying the utility/cost/loss function to the predicted risk.

Depends on what you want to achieve from the classification?

Say it is cancer v/s non cancer, then detecting cancer is vital. However since non-cancer will form majority of your data the classifier can essentially send all cases to non-cancer class and get very high accuracy. But we can't afford that, so we essentially down sample non-cancer cases, essentially moving the decision boundary away from cancer region into the non-cancer region.

Even in use cases where accuracy is our only aim, balancing can be essential if the test time balance is expected to be different from train time.

For example say you want to classify mangoes and oranges, you have a training dataset with 900 mangoes and 30 oranges, but you expect to deploy it in a marketplace with equal mangoes and oranges, then ideally you should sample in the expected sample ratio to maximize accuracy.

  • That is what I understood from the lectures I had. However, I don't understand when balancing can be bad, as this blog post suggests. Why would it ever be bad to balance, if sufficient data points remain for each class? – Zelphir Aug 19 '16 at 13:53
  • Sorry, but in your analogy, what does the market fruit distribution has to do with model accuracy? You either learned to separate mangoes from oranges, or not. In other words, you should be able to deploy the same model on a orange-only or mangoes-only market. – Fernando Mar 2 '17 at 20:43
  • 3
    But the problem with the cancer example is to view it as classification, it should be treated as risk estimation. Then the apparent problem with unbalanced classes disappears, see stats.stackexchange.com/questions/127042/… – kjetil b halvorsen Jun 2 '17 at 11:41

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.