When should I balance classes in a training data set?

Question

I had an online course, where I learned, that unbalanced classes in the training data might lead to problems, because classification algorithms go for the majority rule, as it gives good results if the unbalance is too much. In an assignment one had to balance the data via undersampling the majority class.

In this blog however, someone claims that balanced data is even worse:

https://matloff.wordpress.com/2015/09/29/unbalanced-data-is-a-problem-no-balanced-data-is-worse/

So which one is it? Should I balance the data or not? Does it depend on the algorithm used, as some might be able to adept to the unbalanced proportions of classes? If so, which ones are reliable on unbalanced data?

Answer 1

The intuitive reasoning has been explained in the blogpost:

If our goal is Prediction, this will cause a definite bias. And worse, it will be a permanent bias, in the sense that we will not have consistent estimates as the sample size grows.

So, arguably the problem of (artificially) balanced data is worse than the unbalanced case.

Balanced data are good for classification, but you obviously loose information about appearance frequencies, which is going to affect accuracy metrics themselves, as well as production performance.

Let's say you're recognizing hand-written letters from English alphabet (26 letters). Overbalancing every letter appearance will give every letter a probability of being classified (correctly or not) roughly 1/26, so classifier will forget about actual distribution of letters in the original sample. And it's ok when classifier is able to generalize and recognize every letter with high accuracy.

But if accuracy and most importantly generalization isn't "so high" (I can't give you a definition - you can think of it just as a "worst case") - the misclassified points will most-likely equally distribute among all letters, something like:

"A" was misclassified 10 times
"B" was misclassified 10 times
"C" was misclassified 11 times
"D" was misclassified 10 times
...and so on

As opposed to without balancing (assuming that "A" and "C" have much higher probabilities of appearance in text)

"A" was misclassified 3 times
"B" was misclassified 14 times
"C" was misclassified 3 times
"D" was misclassified 14 times
...and so on

So frequent cases will get fewer misclassifications. Whether it's good or not depends on your task. For natural text recognition, one could argue that letters with higher frequencies are more viable, as they would preserve semantics of the original text, bringing the recognition task closer to prediction (where semantics represent tendencies). But if you're trying to recognize something like screenshot of ECDSA-key (more entropy -> less prediction) - keeping data unbalanced wouldn't help. So, again, it depends.

The most important distinction is that the accuracy estimate is, itself, getting biased (as you can see in the balanced alphabet example), so you don't know how the model's behavior is getting affected by most rare or most frequent points.

P.S. You can always track performance of unbalanced classification with Precision/Recall metrics first and decide whether you need to add balancing or not.

---

EDIT: There is additional confusion that lies in estimation theory precisely in the difference between sample mean and population mean. For instance, you might know (arguably) actual distribution of English letters in the alphabet $p(x_i | \theta)$, but your sample (training set) is not large enough to estimate it correctly (with $p(x_i | \hat \theta)$). So in order to compensate for a $\hat \theta_i - \theta_i$, it is sometimes recommended to rebalance classes according to either population itself or parameters known from a larger sample (thus better estimator). However, in practice there is no guarantee that "larger sample" is identically distributed due to risk of getting biased data on every step (let's say English letters collected from technical literature vs fiction vs the whole library) so balancing could still be harmful.

This answer should also clarify applicability criteria for balancing:

The class imbalance problem is caused by there not being enough patterns belonging to the minority class, not by the ratio of positive and negative patterns itself per se. Generally if you have enough data, the "class imbalance problem" doesn't arise

As a conclusion, artificial balancing is rarely useful if training set is large enough. Absence of statistical data from a larger identically distributed sample also suggests no need for artificial balancing (especially for prediction), otherwise the quality of estimator is as good as "probability to meet a dinosaur":

What is the probability to meet a dinosaur out in the street?

1/2 you either meet a dinosaur or you do not meet a dinosaur

Answer 2

Consistent with @kjetil-b-halvorsen's comment, the rapid adoption of machine learning has confused researchers about prediction vs. classification. As I described in more detail here, classification is only appropriate in a minority of cases. When the outcome is rare (or too common), probabilities are everything because in that case one can only reasonably speak about tendencies, not about predicting individual occurrences.

In statistics, we learned a while back that any method that requires one to exclude some of the data is highly suspect. So the goal of balancing outcomes is misplaced. Prediction of tendencies (probabilities) does not require it. And once you estimate a probability you can make an optimal decision by applying the utility/cost/loss function to the predicted risk.

Answer 3

Depends on what you want to achieve from the classification?

Say it is cancer v/s non cancer, then detecting cancer is vital. However since non-cancer will form majority of your data the classifier can essentially send all cases to non-cancer class and get very high accuracy. But we can't afford that, so we essentially down sample non-cancer cases, essentially moving the decision boundary away from cancer region into the non-cancer region.

Even in use cases where accuracy is our only aim, balancing can be essential if the test time balance is expected to be different from train time.

For example say you want to classify mangoes and oranges, you have a training dataset with 900 mangoes and 30 oranges, but you expect to deploy it in a marketplace with equal mangoes and oranges, then ideally you should sample in the expected sample ratio to maximize accuracy.

Answer 4

When your data is balanced you can prefer to check the metric accuracy. But when such a situation your data is unbalanced your accuracy is not consistent for different iterations. You need to concentrate more metrics like Precision(PPR), Recall(sensitivity). This two metrics should be balanced when compare. Also you should have to check F1-Score which is harmonic mean of Precision and recall. This is applicable for all the machine learning algorithms