Unbalanced data on fire for a binary classifier

Question

I have a lot of training data from which I want to build a binary classifier, but the classes are highly unbalanced, 97% in one class, 3% in the other (even though, in absolute terms, I still have a lot of data, thousands of observations) in each class.

I know there are a lot of fancy algorithms to deal with unbalanced classes, but for lack of time I just want to know in my scenario: Is it safe to throw away a lot of the labelled data from the larger class, so that I have roughly the same amount of labels of both types and then split this dataset into a training(+crossvalidation) and test dataset?

Answer 1

Class imbalance is almost certainly not a problem for proper statistical methods. Therefore, throwing away precious data just to fix a non-problem seems like the worst approach one can take. Additionally, by throwing away many instances of the majority category, you alter the prior probability of membership in that category, which will affect the posterior (predicted) probability through Bayes' theorem. It is possible to fix this, but why would you throw away precious data to solve a non-problem just to have to fix a real problem later that only exists because you tried to solve a non-problem?

From King and Zeng (2001), it can make sense to downsample at the stage of collecting data. If you have too large of a data set to run it on your hardware, it might make sense to apply ideas like those of King and Zeng (2001), giving your hardware a break without discarding especially precious observations from the minority category (and then calibrating the predictions to account for this). However, getting rid of precious data just to solve a non-problem seems like a poor approach.

REFERENCE

King, Gary, and Langche Zeng. "Logistic regression in rare events data." Political analysis 9.2 (2001): 137-163.