I fitted a binary logit model with unbalanced data which were oversampled using SMOTE. This gave an excellent ROC curve but very poor adequacy - the zero hypothesis of adequacy was rejected by Hosmer-Lemeshow test and le Cessie – van Houwelingen – Copas – Hosmer unweighted sum of squares test. However, the logit model for the original data (without oversampling) had very good adequacy statistics (but mediocre classification properties).

Are there any ways to oversample data without ruining adequacy statistics for a binary logit/probit model?

This may be due to your intercept term, which reflects the base rates. You trained the model on base rates that don’t reflect reality, so it needs to be corrected. Here is a paper discussing the problem and solutions king & zeng

  • 3
    SMOTE is an invalid statistical technique. Under no circumstances should you use it. Never under- or over-sample data you have already sampled. Over and under-sampling is only appropriate when obtaining the data in the first place, if cost is an issue. The development of SMOTE was an attempt to deal with improper accuracy scoring rules. Two wrongs do not make a right. See fharrell.com/post/class-damage . – Frank Harrell Apr 30 at 19:07

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.