1
$\begingroup$

I am working on an imbalance dataset with a 98:2 ratio (1M record in the majority class and 20K in the minority class) I am planning to run my model for 30 folds,

  • I tried with stratified K folds but here again, each fold will have an imbalance problem again
  • I don't want to use SMOTE analysis as it creates data loss or Overfitting

which cross-validation techniques or hyperparameters should I use so that all Minority classes are included in each fold?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I think including minority class in all folds and answering imbalance problem are two separate issues.

If you use StratifiedKFold with 30 folds, that means you will train your model on 29/30 ~ 97% of your majority class and your minority data. Your test data will be ~3% of each class. That is a good approach but you'll have to face the imbalance problem, true. As you said, SMOTE is one approach, you could also use random subselection of majority class, or the class_weight hyperparameter of some classifiers, or ensemble approaches, or other approaches...

So, combining StratifiedKFold + imbalance handling techniques seems the right approach to me.

About:

so that all Minority classes are included in each fold?

I am not sure to understand what you mean by all minority classes as you said there is only one minority class, but note that if you include all samples from minority class in each fold, you will train and test your model on the same data. You definitely don't want to do that.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Please feel free to comment. I might have misunderstood the question, and I'd be glad to update my answer, but downvoting without commenting is not really helpful... $\endgroup$
    – etiennedm
    Jan 25 at 10:37
  • 1
    $\begingroup$ There is sample code for stratified cv here: machinelearningmastery.com/… $\endgroup$
    – G. Grothendieck
    Jan 25 at 10:51
  • $\begingroup$ what i though was, Majority/Minority = 33, so my plan was to create 33 folds and in each fold include all minority class, so that in each fold Majority=minority class and hence training the model instead of doing SMOTE $\endgroup$
    – Shiv948
    Jan 25 at 12:40
  • 1
    $\begingroup$ If you do so and use your folds in the "common" way, you will use k-1 folds for training, so 32. If each fold contains all minority samples, you will have each sample from minority class duplicated 32 times (one from each fold). Plus, (and more importantly), you will test on the remaining fold that also contains all minority samples, so you will train and test a model partially on the same data (the minority class). That is wrong because you might not learn but only remember. $\endgroup$
    – etiennedm
    Jan 25 at 13:25
  • 1
    $\begingroup$ One solution that might be close to what you wanted to do and that might work is to apply first a StratifiedKFold. Then with your training data from the k-1 fold, let's say you have 5 times more of majority class. You could fit 5 different models that each trains on all minority class + 1/5th of majority class. Finally you combined them via VotingClassifier or another ensemble method. But again, this is one way of dealing with imbalance data. Other approaches might be better. $\endgroup$
    – etiennedm
    Jan 25 at 13:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.