Class imbalance: training set is balanced but test set is imbalanced, how to train?

Question

I have a huge dataset, say around 100M data points, with a class imbalance of 1 positive for every 100 negatives. It is very difficult to train on the entire dataset, so I tend to undersample the negatives such that the training data becomes balanced (1:1). But the test set remains imbalanced to reflect the real life nature of the data. FYI, I use a simple feedforward neural network

How do I go about training in such a combination? I'd use class weights during training but I'm thinking it might overpredict on the test set? Moreover how do I evaluate this model with AUC and AUPRC, do I need to use some form of weighting?

Answer 1

The primary effect on a model of downsampling like this is a shift in the predicted log-odds. This is rigorously shown for logistic regression, see https://stats.stackexchange.com/a/68726/232706; for other models, I've observed the same effect (though I don't do a lot of neural nets). Assuming that's really true in your case, you can "fix" the probability estimates by adding the adjustment term listed in the above link. Note too that such a monotonic adjustment should not affect AUROC or AUPRC at all.

Using class weights in training instead should produce a very similar effect; see https://datascience.stackexchange.com/a/58899/55122

You have "plenty" (difficult to say without more context, but 1M is a lot) of positive examples, so the suggestion to sample without affecting class balance given by @StephanKolassa in a comment may also be fine. In other contexts where the positive class is so small that you wouldn't want to throw away any information from them, I think downsampling the giant negative class is fine (and note that Scortchi in the first link mentions exactly this case).

Answer 2

Before, you should use stratify method for data sampling after that You have to give weight to your model