I'm working with a dataset containing several classes. The largest class has over 500 samples, and the smallest classes have fewer than 10 samples. I know that you should perform upsampling inside the cross-validation loop to prevent data-leakage between folds... however this results in awful balanced-accuracy because there are so few samples for those tiny classes.

Would it be valid to perform a tiny amount of upsampling using a technique like SMOTE to lets say increase the number of samples in the smallest classes to something like 20 samples prior to cross validation and them perform further upsampling with SMOTE normally inside the cross-validation loop?

  • 2
    $\begingroup$ Unbalanced classes are almost certainly not a problem, and oversampling will not solve a non-problem: Are unbalanced datasets problematic, and (how) does oversampling (purport to) help? $\endgroup$ – Stephan Kolassa Mar 31 at 14:39
  • 2
  • 1
    $\begingroup$ @StephanKolassa I hope you're in the process of making a bot to automate your response to class imbalance questions! But since it is relevant, can you confirm specifically that SMOTE is not useful? Or are there cases where it is? I can't figure out what SMOTE's utility is (stats.stackexchange.com/questions/285231/…) but likewise have not found a clear statement that it is actually useless. Additionally, it would be surprising to me that a commonly known and utilized algorithm actually serves no constructive purpose. $\endgroup$ – Ryan Volpi Mar 31 at 15:15
  • 2
    $\begingroup$ @RyanVolpi: that bot would indeed be an interesting idea... to be honest, I have not looked into SMOTE specifically, simply because I am indeed convinced that it is an attempt to solve a non-problem. If you find interesting evidence against this preconception of mine, please do tell me. In the meantime, unfortunately it is not at all uncommon for a commonly known and utilized theory to serve no constructive purpose. I'll start calling upsampling the humor theory of classification. Let's hope it won't take 2000 years to fall out of favor. $\endgroup$ – Stephan Kolassa Mar 31 at 15:21
  • 2
    $\begingroup$ I would recommend (see the threads above) not using a classifier that outputs a hard classification, but using probabilistic classifications. And evaluating these not using accuracy (which also necessitates a threshold, which is another problem, see the threshold thread above), but proper scoring rules. Per the very first question linked above, there is no bias problem any more - just possibly very small predicted class memberships, which is precisely as it should be. $\endgroup$ – Stephan Kolassa Mar 31 at 15:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.