0
$\begingroup$

I'm building a Random Forest model over an unbalaced 4 class dataset. So far I understood how to use oversampling and train my model. My doubt was about when to perform Oversampling.

I've already seen a lot of questions about oversampling before or after the train/test split, and I already know that the best way is to split into train/test before and then apply oversampling.

My doubt regards this second scenario (oversamplig after splitting).

Suppose that I have already splitted my dataset in train and test with a percentage of 80%-20% and I get my X_train, y_train, X_test, y_test data.

Now I'm going to perform (for example) cross validation over my X_train in order to estimate my validation error. For example (using Python) I could have something like:

from sklearn.model_selection import cross_val_score
from imblearn.pipeline import Pipeline, make_pipeline
from sklearn.ensemble import RandomForestClassifier
from imblearn.over_sampling import SMOTE

imba_pipeline = make_pipeline(SMOTE(sampling_strategy='auto', k_neighbors=10,random_state = SEED), 
                              RandomForestClassifier(n_estimators=200, bootstrap=False,  min_samples_leaf=2, min_samples_split=2, max_depth=14, random_state=SEED, class_weight='balanced',max_features = 'sqrt'))

scores=cross_val_score(imba_pipeline, X_train, y_train, scoring='accuracy', cv=10)
print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))

Now I'm happy with my crossvalidation score and I want to train my final model.

Should I retrain it over the X_train oversampled? So basically I'll do something like:

sm = SMOTE(sampling_strategy='auto', k_neighbors=10,random_state = SEED)
X_train_upsample, y_train_upsample = sm.fit_sample(X_train, y_train)
clf=RandomForestClassifier(n_estimators=200, bootstrap=False,  min_samples_leaf=2, min_samples_split=2, max_depth=14, random_state=SEED, class_weight='balanced',max_features = 'sqrt')).fit(X_train_upsample, y_train_upsample)

Or is it a bad idea?

What if I performed crossvalidation on the already oversampled dataset? So, instead of oversampling each single fold, I have something like:

sm = SMOTE(sampling_strategy='auto', k_neighbors=10,random_state = SEED)
X_train_upsample, y_train_upsample = sm.fit_sample(X_train, y_train)

clf=RandomForestClassifier(n_estimators=200, bootstrap=False,  min_samples_leaf=2, min_samples_split=2, max_depth=14, random_state=SEED, class_weight='balanced',max_features = 'sqrt'))

scores=cross_val_score(clf, X_train_upsample, y_train_upsample, scoring='accuracy', cv=10)
$\endgroup$

1 Answer 1

1
$\begingroup$

I already know that the best way is to split into train/test before and then apply oversampling.

No, it isn't. Are unbalanced datasets problematic, and (how) does oversampling (purport to) help? Answer: they aren't, and it doesn't.

(See here for a motivation for short answers. Longer answers are always welcome. See here for a general motivation for an answer that essentially says that "the question is wrong".)

$\endgroup$
13
  • 1
    $\begingroup$ The post seems more than reasonable. But why there is such a huge amount of post and papers talking about class unbalance? I could point out plenty of resources which talk about class unbalance and they solve it with (for example) oversamplig/undersampling and weighting methods. Who should I trust? For example, class unbalance in Image Classification problems, is definitely a problem, am i wrong? $\endgroup$ Sep 19, 2020 at 15:30
  • 2
    $\begingroup$ To be honest, I believe that a lot of noise about oversampling comes from people who, sorry, don't know statistics and don't know how to think statistically, which in turn is mainly driven by the fact that much of "ML" has been appropriated by computer scientists, not the statistical community. $\endgroup$ Sep 19, 2020 at 15:46
  • 2
    $\begingroup$ On Cross Validated, you are likely to get more statistical answers. I submit that our answers are better than oversampling. If you disagree, I would be very interested in your arguments, because I have never seen a serious one beyond "accuracy does weird things with unbalanced data", which statisticians have a (IMO) better answer to than oversampling. $\endgroup$ Sep 19, 2020 at 15:46
  • 2
    $\begingroup$ @MattiaSurricchio here's an example of a good use for some type of F-score (there's a whole family, based on tradeoffs between precision and recall). In that case one is looking for objects, and there are no true-negative objects, just an object-free background. Some text-classification schemes are similar: you primarily want to identify some texts on a particular topic out of a large corpus and don't care much about off-topic texts in the corpus so long as you find enough on the topic. So "usefulness" is very application-dependent. $\endgroup$
    – EdM
    Sep 19, 2020 at 16:53
  • 2
    $\begingroup$ Use probabilistic class membership predictions. Evaluate these using proper scoring rules, like the Brier or the log score. These will provably (!) draw you towards correct probabilistic classifications (that's what "proper" means for scoring rules). You can find a few links here. $\endgroup$ Sep 20, 2020 at 5:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.