# Should I balance the classifier train/test set, if metrics is Precision/Recall (F1 score)?

I want to train a classifier on an unbalanced data set. Proportions of classes C0/C1 are 65/35. Importantly, the success metrics is F1_score. In other words, the proper classification of class 1 (precision, recall) is important while identification of class 0 (specificity, true negatives rate) is far less important.

Most classifiers expect balanced sets (50/50). So the literature suggests to start with under/oversampling the train data. But what about the test set, and validation set? I think those should rather reflect reality. So eventhough train set would be rebalanced (50/50), I understand that both test and validation sets should keep the 65/35 ratio of classes.

And here lies the problem: the above does not seem to make sense, if we focus on F1 (Precision+Recall). Suppose I train my classifier on that 50/50 train sets. If my test set was also balanced 50/50, I would maybe get Accuracy = Precision = Recall = F1 = 0.8 (just as an example). But ... on the 65/35 test set, the results will be immediately worse: because class C0 is overrepresented in reality, thus the ratio of Negatives will be high, and so the False Negatives will grow proportionally. This will badly impact Precision and F1. In the result, my F1_score will be 0.5 or so. So... I wasted all effort because I trained the classifier on artificial data which did not reflect reality. This problem is described in more detail in the article linked below:

https://towardsdatascience.com/handling-imbalanced-datasets-in-machine-learning-7a0e84220f28 In summary, "we show the wrong proportions of the two classes to the classifier during the training. The classifier learned this way will then have a lower accuracy on the future real test data."

I share the author's observation, but I don't know what is the way out. The author suggests to maybe not balance the train, which I am also doubtful of, as this brings in other obvious problems. To summarize my question. If the metric is F1_score, and if the real data in unbalanced (imbalanced), then how to proceed to achieve optimal F1_score of the trained model in the real scenario:

1. should we really balance the train set (50/50)?
2. if so, should we also balance the test set (50/50)?
3. if so, should we also balance the validation set (50/50)?

What's your best practice? also, are there other techniques of dealing with this situation, I should be aware of?

• None of the above...use proper scoring rules. The flaws with threshold-based metrics like accuracy, precision, recall, and $F_1$ score are covered extensively in Cross Validated. Start here: stats.stackexchange.com/questions/312780/…. Check out the blog posts by Frank Harrell that Kolassa links, too.
– Dave
Jul 21, 2020 at 10:36
• @Dave 's comment helped. I went through extensive lecture and just posted a separate answer with the findings summary: what can be done instead of balancing, which is generally not recommended. But I still fail to understand one thing. There are quite a lot of authors, blog posts and literature that recommend balancing. Are they all wrong? Or do I still miss something. Are there particular situations, in which artificial balancing of the train set is actually recommended? Jul 29, 2020 at 9:49
• Frank Harrell, biostatistician at Vanderbilt University and member of Cross Validated, has two blog posts on this topic: 1 2.
– Dave
Jun 17, 2021 at 17:43