26
$\begingroup$

I had an online course, where I learned, that unbalanced classes in the training data might lead to problems, because classification algorithms go for the majority rule, as it gives good results if the unbalance is too much. In an assignment one had to balance the data via undersampling the majority class.

In this blog however, someone claims that balanced data is even worse:

https://matloff.wordpress.com/2015/09/29/unbalanced-data-is-a-problem-no-balanced-data-is-worse/

So which one is it? Should I balance the data or not? Does it depend on the algorithm used, as some might be able to adept to the unbalanced proportions of classes? If so, which ones are reliable on unbalanced data?

$\endgroup$
26
$\begingroup$

The intuitive reasoning has been explained in the blogpost:

If our goal is Prediction, this will cause a definite bias. And worse, it will be a permanent bias, in the sense that we will not have consistent estimates as the sample size grows.

So, arguably the problem of (artificially) balanced data is worse than the unbalanced case.

Balanced data are good for classification, but you obviously loose information about appearance frequencies, which is going to affect accuracy metrics themselves, as well as production performance.

Let's say you're recognizing hand-written letters from English alphabet (26 letters). Overbalancing every letter appearance will give every letter a probability of being classified (correctly or not) roughly 1/26, so classifier will forget about actual distribution of letters in the original sample. And it's ok when classifier is able to generalize and recognize every letter with high accuracy.

But if accuracy and most importantly generalization isn't "so high" (I can't give you a definition - you can think of it just as a "worst case") - the misclassified points will most-likely equally distribute among all letters, something like:

"A" was misclassified 10 times
"B" was misclassified 10 times
"C" was misclassified 11 times
"D" was misclassified 10 times
...and so on

As opposed to without balancing (assuming that "A" and "C" have much higher probabilities of appearance in text)

"A" was misclassified 3 times
"B" was misclassified 14 times
"C" was misclassified 3 times
"D" was misclassified 14 times
...and so on

So frequent cases will get fewer misclassifications. Whether it's good or not depends on your task. For natural text recognition, one could argue that letters with higher frequencies are more viable, as they would preserve semantics of the original text, bringing the recognition task closer to prediction (where semantics represent tendencies). But if you're trying to recognize something like screenshot of ECDSA-key (more entropy -> less prediction) - keeping data unbalanced wouldn't help. So, again, it depends.

The most important distinction is that the accuracy estimate is, itself, getting biased (as you can see in the balanced alphabet example), so you don't know how the model's behavior is getting affected by most rare or most frequent points.

P.S. You can always track performance of unbalanced classification with Precision/Recall metrics first and decide whether you need to add balancing or not.


EDIT: There is additional confusion that lies in estimation theory precisely in the difference between sample mean and population mean. For instance, you might know (arguably) actual distribution of English letters in the alphabet $p(x_i | \theta)$, but your sample (training set) is not large enough to estimate it correctly (with $p(x_i | \hat \theta)$). So in order to compensate for a $\hat \theta_i - \theta_i$, it is sometimes recommended to rebalance classes according to either population itself or parameters known from a larger sample (thus better estimator). However, in practice there is no guarantee that "larger sample" is identically distributed due to risk of getting biased data on every step (let's say English letters collected from technical literature vs fiction vs the whole library) so balancing could still be harmful.

This answer should also clarify applicability criteria for balancing:

The class imbalance problem is caused by there not being enough patterns belonging to the minority class, not by the ratio of positive and negative patterns itself per se. Generally if you have enough data, the "class imbalance problem" doesn't arise

As a conclusion, artificial balancing is rarely useful if training set is large enough. Absence of statistical data from a larger identically distributed sample also suggests no need for artificial balancing (especially for prediction), otherwise the quality of estimator is as good as "probability to meet a dinosaur":

What is the probability to meet a dinosaur out in the street?

1/2 you either meet a dinosaur or you do not meet a dinosaur

$\endgroup$
  • 3
    $\begingroup$ I think besides the explanation of the issue, the important take-away from this answer is that one should try unbalanced first and check its results and only if necessary do the balancing and check its result. +1 $\endgroup$ – Zelphir Mar 28 '17 at 11:35
  • $\begingroup$ So, in other words, with evenly distributed classes to the training subset the model will loose its accuracy in unseen data, right? But, in the opposite case, where you try to randomly extract entries of a dataset for your training/testing subsets, will your classifier perform better? $\endgroup$ – Christos K. Dec 23 '18 at 18:49
  • $\begingroup$ @ChristosK. As many stated, when you see the problem as classification, it’s hard to reason about prediction. In any case, if you remove bias (sample “randomly”) - you need a bigger sample to improve performance. It’s just “usually” sample is big enough to preserve semantics, so overbalancing would only hurt and act like regularizing hammer that “flattens” everything without proper consideration. Also, as dinosaur metaphor suggests, “balanced” doesn’t mean “even” - you do proper balancing only when you know that some probabilities are misrepresented in a “random” sample. $\endgroup$ – dk14 Dec 23 '18 at 20:03
  • 1
    $\begingroup$ @ChristosK. Thanks for some clarifications. It’s not the same what I meant but the approach is very similar. Usual recommendation for applicability of k-fold is to do it when your initial sample is “kinda small”. Not sure, but folding shouldn’t hurt anyways - it’s just it takes more runs, and the less you care about prediction, the less you care about generalization/performance as tautological as it sounds :). But overall - k-fold means less bias essentially. $\endgroup$ – dk14 Dec 23 '18 at 20:40
  • 1
    $\begingroup$ @ChristosK. Oh, and as a warning, spam/not-spam ratio might be a non-stationary random variable on its own. With all those “fake news”, “russian trolls” and other stuff I’d be careful about such assumptions - ratio could be biased too. You might want to estimate PrecisionRecall on your classifiers first, if something is under-sampled - I’d rather collect/generate(?) more data. $\endgroup$ – dk14 Dec 23 '18 at 20:56
17
$\begingroup$

Consistent with @kjetil-b-halvorsen's comment, the rapid adoption of machine learning has confused researchers about prediction vs. classification. As I described in more detail here, classification is only appropriate in a minority of cases. When the outcome is rare (or too common), probabilities are everything because in that case one can only reasonably speak about tendencies, not about predicting individual occurrences.

In statistics, we learned a while back that any method that requires one to exclude some of the data is highly suspect. So the goal of balancing outcomes is misplaced. Prediction of tendencies (probabilities) does not require it. And once you estimate a probability you can make an optimal decision by applying the utility/cost/loss function to the predicted risk.

$\endgroup$
1
$\begingroup$

Depends on what you want to achieve from the classification?

Say it is cancer v/s non cancer, then detecting cancer is vital. However since non-cancer will form majority of your data the classifier can essentially send all cases to non-cancer class and get very high accuracy. But we can't afford that, so we essentially down sample non-cancer cases, essentially moving the decision boundary away from cancer region into the non-cancer region.

Even in use cases where accuracy is our only aim, balancing can be essential if the test time balance is expected to be different from train time.

For example say you want to classify mangoes and oranges, you have a training dataset with 900 mangoes and 30 oranges, but you expect to deploy it in a marketplace with equal mangoes and oranges, then ideally you should sample in the expected sample ratio to maximize accuracy.

$\endgroup$
  • $\begingroup$ That is what I understood from the lectures I had. However, I don't understand when balancing can be bad, as this blog post suggests. Why would it ever be bad to balance, if sufficient data points remain for each class? $\endgroup$ – Zelphir Aug 19 '16 at 13:53
  • $\begingroup$ Sorry, but in your analogy, what does the market fruit distribution has to do with model accuracy? You either learned to separate mangoes from oranges, or not. In other words, you should be able to deploy the same model on a orange-only or mangoes-only market. $\endgroup$ – Fernando Mar 2 '17 at 20:43
  • 3
    $\begingroup$ But the problem with the cancer example is to view it as classification, it should be treated as risk estimation. Then the apparent problem with unbalanced classes disappears, see stats.stackexchange.com/questions/127042/… $\endgroup$ – kjetil b halvorsen Jun 2 '17 at 11:41
1
$\begingroup$

When your data is balanced you can prefer to check the metric accuracy. But when such a situation your data is unbalanced your accuracy is not consistent for different iterations. You need to concentrate more metrics like Precision(PPR), Recall(sensitivity). This two metrics should be balanced when compare. Also you should have to check F1-Score which is harmonic mean of Precision and recall. This is applicable for all the machine learning algorithms

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.