1
$\begingroup$

I have an imbalanced dataset and I'm trying to predict a binary target. The minority class amounts to approximately 0.4% of all observations (60 million observations from which 250K belong to the minority class).

I undersampled the majority class and oversampled the minority class so that the mean of my target column is now 23%. Would it be correct to train-test split this balanced dataset?

In other words, Should I:

  1. train-test split the data after rebalancing; or
  2. assign a subset of the original dataset for testing, assign a subset of the original dataset for training and rebalance the training dataset?
Improve this question
$\endgroup$
2
  • $\begingroup$ How many observations are there? It's possible that you don't have enough cases to make any test/train split reliable. See this post by Frank Harrell. Also, that type of over/undersampling can lead to problems. Please add the information about your sample size and the hypothesis you want to test by editing the question, as comments are easy to overlook and can be deleted. $\endgroup$
    – EdM
    Commented Mar 7 at 18:05
  • $\begingroup$ "I undersampled the majority class and oversampled the minority class so that the mean of my target column is now 23%" why 23%? As you have a lot of data, class imbalance is unlikely to be problematic. I suspect the real issue here is that misclassification costs are unequal (the minority class is "more important" in some respect). The thing to do is to work out what the misclassification costs might resonably be and use cost-sensitive learning/Bayesian decision theory/minimum risk classification. $\endgroup$
    – Dikran Marsupial
    Commented Mar 10 at 11:20

1 Answer 1

Reset to default
1
$\begingroup$

An imbalanced data set, especially, if the minority class is the one you are interested in, stays imbalanced. Sampling methods in general do not help, because you are not allowed to test your model also on up or downsampled data.

In detail, even if you up or downsample the training data, you need to test your algorithm on the pure imbalanced data.

If you are worried regarding the split, then a Stratified Kfold may help you: https://stackoverflow.com/questions/65318931/stratifiedkfold-vs-kfold-in-scikit-learn

If you are worried about your precision/recall, then you can adjust the treshold for the positive class after the training, so that minority classes may be easier identified afterwards. This is called post-hoc.

You may also use Negative Mining. This can be used with classifiers, which accept sample weights. In detail, you would punish misclassifications of false positives (so to speak your model thinks an observation is the minority class but it isn't). This requires a model for training, to get the misclassifications, and then a second model with updated weights as a parameter.

The latter is done post-hoc and during training.

In summary, most of the time, you do nothing. You can adjust the threshold or sample weights during training. Up or downsampling like SMOTE is not useful, as EdM has already pointed out.

NegativeMining also appears in DeepLearning, for those who are interested: https://finetuner.jina.ai/advanced-topics/negative-mining/?ref=jina-ai-gmbh.ghost.io

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.