I have a dataset with an event rate of less than 0.3 percent. To improve the modeling results, I did some oversampling using SMOTE.

I initially oversampled so that the event rate increases 10 times to 3 percent. But that doesn't feel right. Are there any restrictions or heuristics on how much we can oversample.

Are there things I need to consider in deciding how much to oversample.

  • $\begingroup$ There's no general rule because it depends on the specific dataset. You can try to perform cross validation using increasing percentages and check whether there is a jump in the performances at some point. Then you can investigate what happens around that range. $\endgroup$ – user289381 Jul 17 '20 at 19:00
  • $\begingroup$ Focus on an appropriate metric first. And consider rebalancing the sample (if indeed needed). $\endgroup$ – usεr11852 Jul 17 '20 at 19:01
  • $\begingroup$ @usεr11852 I am using F1 scores to evaluate the model. Post oversampling I get good very different F1 score on train and test but consistent results for recall score on train and test. $\endgroup$ – Clock Slave Jul 17 '20 at 19:06
  • $\begingroup$ @ping I'll try that $\endgroup$ – Clock Slave Jul 17 '20 at 19:07
  • $\begingroup$ Just to be clear, we must never use our oversampled set for testing. $F_1$ is fine but do note that it does not account for True Negatives at all. Maybe using a scoring rule like AUC-ROC or Brier score is more informative. $\endgroup$ – usεr11852 Jul 17 '20 at 19:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.