4
$\begingroup$

I have a huge dataset, say around 100M data points, with a class imbalance of 1 positive for every 100 negatives. It is very difficult to train on the entire dataset, so I tend to undersample the negatives such that the training data becomes balanced (1:1). But the test set remains imbalanced to reflect the real life nature of the data. FYI, I use a simple feedforward neural network

How do I go about training in such a combination? I'd use class weights during training but I'm thinking it might overpredict on the test set? Moreover how do I evaluate this model with AUC and AUPRC, do I need to use some form of weighting?

$\endgroup$
17

2 Answers 2

3
$\begingroup$

The primary effect on a model of downsampling like this is a shift in the predicted log-odds. This is rigorously shown for logistic regression, see https://stats.stackexchange.com/a/68726/232706; for other models, I've observed the same effect (though I don't do a lot of neural nets). Assuming that's really true in your case, you can "fix" the probability estimates by adding the adjustment term listed in the above link. Note too that such a monotonic adjustment should not affect AUROC or AUPRC at all.

Using class weights in training instead should produce a very similar effect; see https://datascience.stackexchange.com/a/58899/55122

You have "plenty" (difficult to say without more context, but 1M is a lot) of positive examples, so the suggestion to sample without affecting class balance given by @StephanKolassa in a comment may also be fine. In other contexts where the positive class is so small that you wouldn't want to throw away any information from them, I think downsampling the giant negative class is fine (and note that Scortchi in the first link mentions exactly this case).

$\endgroup$
0
$\begingroup$

Before, you should use stratify method for data sampling after that You have to give weight to your model

$\endgroup$
2
  • 2
    $\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$
    – utobi
    Nov 13, 2022 at 6:10
  • 1
    $\begingroup$ Welcome to Cross Validated! I second what @utobi wrote. This is the beginning of what could be a quality answer but really needs details added, especially considering the presence of another answer and multiple comments that discuss class imbalance issues extensively. $\endgroup$
    – Dave
    Nov 13, 2022 at 6:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.