2
$\begingroup$

I have an imbalanced (two-class) classification dataset, based on which I am trying to train and cross-validate a classifier.

During the process of the k-fold cross-validation, I set aside the test subsets before I oversample the remaining (training) subsets. I also standardize (zscore) the training subsets, and I store the Mu and Sigma to be later used in standardizing the test subsets.

The problem is that I do not know whether I should do the oversampling first or the standardization first! Any recommendations/suggestions would be appreciated.

$\endgroup$

1 Answer 1

3
$\begingroup$

The question is, why do you oversample?

  • If you have different relative frequencies in your data than you expect in the real application and oversampling is to correct this - then oversampling should be done first (or, to put it differently, you calculated weighted mean and standard deviation, and train a classifier for the corrected prior probabilities).

  • If you oversample "only" because you have imbalanced classes and try to generate a balanced data set, then IMHO you need to think twice whether this oversampling is good idea at all: there's no use in a classifier that is optimized for balanced classes if in reality the classes are just as imbalanced as your data.

  • I take from your question that you are already aware that splitting of training and test sets needs to be done first, so cases are independent.

$\endgroup$
4
  • $\begingroup$ Interesting... in the second point (oversampling to generate a balanced dataset when in reality the classes are imbalanced), what would you suggest instead of oversampling then? Because i have exactly that problem and i have been using SMOTE to oversample for quite a time now (and getting really good results from my classifier). $\endgroup$
    – Murilo
    May 26, 2022 at 13:08
  • $\begingroup$ @Murilo: do you get the good performance for similarly oversampled test data, or (also) for test data following the imbalanced real-world application distribution? What I'd suggest in that case is to use a classifier that works well with imbalanced data. (E.g., I often meet "imbalance is a problem" where one-class classifiers would really be more appropriate than discriminative/binary classification) $\endgroup$ May 26, 2022 at 13:33
  • $\begingroup$ Let me explain a bit more. I first normalize my features between 0 and 1 (they are really in different scale). After that, i use SMOTE to oversample the minority class. Finally, i split the data into train/test. During training, the train data is again split into train/validation (so i change my hyperparameteres if the validation is not good). Finally, i use the test set to evaluate my model one last time. And answering your question, yes, i get a similar good performance in training, validation and test data. If i try to use my dataset without SMOTE, i get really bad results. $\endgroup$
    – Murilo
    May 26, 2022 at 13:40
  • $\begingroup$ @Murilo: by first SMOTEing before train/test split, you create a) a severe data leak between training (including validation/optimization) and test sets and b) you do not test on data from the application distribution but on the artificial distribution created by SMOTEing. I.e., you are likely severely overfitting. You need to partition off your test set before even normalizing (and you need to make sure that your splitting accounts for any structure in your "raw" data such as (nearly)repeated measurements etc. that could cause further data leaks). Note that OP explicitly says they split first. $\endgroup$ May 26, 2022 at 17:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.