Training data size requirements for imbalanced class classification

Question

Our team is planning to start data collection for an imbalanced binary classification task. One in 30 cases is positive class and the rest 29 of the 30 cases are negative class. We have around 20 features (both categorical and numeric). We are initially planning to collect around 60,000 data points so that we will end up with 2000 positive classes.

Can you please comment if the size of the training data looks alright for the imbalanced classification task?

I understand that the numbers also depend on the finer details like the values of the feature matrix and the type of the classifier. But if we want to have an approximate ballpark number for the data set size, what number would you recommend if we are aiming for an AUC of 0.9? And are there any guidelines for the size of the dataset?

Many thanks

Answer 1

Your ability to get an AUC of 0.9 will depend on how well the predictors you have actually predict class membership. No amount of extra data can get around that fundamental constraint.

A frequent rule of thumb is to make sure that you have at least 15 members of the minority class per predictor you are evaluating. A "predictor" here includes each level of a categorical variable beyond the first, and each interaction term that you include. So you should have no problem in building a model for these data, without overfitting, via a flexible logistic regression that incorporates splines for continuous predictors and several interaction terms, or a boosted tree that allows for interactions with deep trees. No promises about an AUC of 0.9, however.

Answer 2

AUC is not changing with class balancing, so unbalanced class won't lower your AUC value.

First, note than many models in sklearns have the option class_weight='balanced' that allows the model dealing with unbalanced classes by itself, so wathever your unbalance ratio is, the result using this option might be better than if you remove it and try to manually balance data.

For algorithm that don't have this option, you can try to manually balance data, trying different combinaisons of SMOTE and Downsampling Majority Class. To me, 1/30 doesn't look critical and I think you can have results if you balance well. For this point, I don't know any guideline method to know how much I have to SMOTE / Downsample, I'd suggest you just to run some tests to find the best SMOTE and Downsample combinaison.