1
$\begingroup$

I have a huge dataset, say around 100M data points, with a class imbalance of 1 positive for every 100 negatives. It is very difficult to train on the entire dataset, so I tend to undersample the negatives such that the training data becomes balanced (1:1). But the test set remains imbalanced to reflect the real life nature of the data. FYI, I use a simple feedforward neural network

How do I go about training in such a combination? I'd use class weights during training but I'm thinking it might overpredict on the test set? Moreover how do I evaluate this model with AUC and AUPRC, do I need to use some form of weighting?

$\endgroup$
17

1 Answer 1

2
$\begingroup$

The primary effect on a model of downsampling like this is a shift in the predicted log-odds. This is rigorously shown for logistic regression, see https://stats.stackexchange.com/a/68726/232706; for other models, I've observed the same effect (though I don't do a lot of neural nets). Assuming that's really true in your case, you can "fix" the probability estimates by adding the adjustment term listed in the above link. Note too that such a monotonic adjustment should not affect AUROC or AUPRC at all.

Using class weights in training instead should produce a very similar effect; see https://datascience.stackexchange.com/a/58899/55122

You have "plenty" (difficult to say without more context, but 1M is a lot) of positive examples, so the suggestion to sample without affecting class balance given by @StephanKolassa in a comment may also be fine. In other contexts where the positive class is so small that you wouldn't want to throw away any information from them, I think downsampling the giant negative class is fine (and note that Scortchi in the first link mentions exactly this case).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.