0
$\begingroup$

I'm building a Random Forest model over an unbalaced 4 class dataset. So far I understood how to use oversampling and train my model. My doubt was about when to perform Oversampling.

I've already seen a lot of questions about oversampling before or after the train/test split, and I already know that the best way is to split into train/test before and then apply oversampling.

My doubt regards this second scenario (oversamplig after splitting).

Suppose that I have already splitted my dataset in train and test with a percentage of 80%-20% and I get my X_train, y_train, X_test, y_test data.

Now I'm going to perform (for example) cross validation over my X_train in order to estimate my validation error. For example (using Python) I could have something like:

from sklearn.model_selection import cross_val_score
from imblearn.pipeline import Pipeline, make_pipeline
from sklearn.ensemble import RandomForestClassifier
from imblearn.over_sampling import SMOTE

imba_pipeline = make_pipeline(SMOTE(sampling_strategy='auto', k_neighbors=10,random_state = SEED), 
                              RandomForestClassifier(n_estimators=200, bootstrap=False,  min_samples_leaf=2, min_samples_split=2, max_depth=14, random_state=SEED, class_weight='balanced',max_features = 'sqrt'))

scores=cross_val_score(imba_pipeline, X_train, y_train, scoring='accuracy', cv=10)
print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))

Now I'm happy with my crossvalidation score and I want to train my final model.

Should I retrain it over the X_train oversampled? So basically I'll do something like:

sm = SMOTE(sampling_strategy='auto', k_neighbors=10,random_state = SEED)
X_train_upsample, y_train_upsample = sm.fit_sample(X_train, y_train)
clf=RandomForestClassifier(n_estimators=200, bootstrap=False,  min_samples_leaf=2, min_samples_split=2, max_depth=14, random_state=SEED, class_weight='balanced',max_features = 'sqrt')).fit(X_train_upsample, y_train_upsample)

Or is it a bad idea?

What if I performed crossvalidation on the already oversampled dataset? So, instead of oversampling each single fold, I have something like:

sm = SMOTE(sampling_strategy='auto', k_neighbors=10,random_state = SEED)
X_train_upsample, y_train_upsample = sm.fit_sample(X_train, y_train)

clf=RandomForestClassifier(n_estimators=200, bootstrap=False,  min_samples_leaf=2, min_samples_split=2, max_depth=14, random_state=SEED, class_weight='balanced',max_features = 'sqrt'))

scores=cross_val_score(clf, X_train_upsample, y_train_upsample, scoring='accuracy', cv=10)
Improve this question
$\endgroup$
1
$\begingroup$

I already know that the best way is to split into train/test before and then apply oversampling.

No, it isn't. Are unbalanced datasets problematic, and (how) does oversampling (purport to) help? Answer: they aren't, and it doesn't.

(See here for a motivation for short answers. Longer answers are always welcome. See here for a general motivation for an answer that essentially says that "the question is wrong".)

Improve this answer
$\endgroup$
13
  • 1
    $\begingroup$ The post seems more than reasonable. But why there is such a huge amount of post and papers talking about class unbalance? I could point out plenty of resources which talk about class unbalance and they solve it with (for example) oversamplig/undersampling and weighting methods. Who should I trust? For example, class unbalance in Image Classification problems, is definitely a problem, am i wrong? $\endgroup$ – Mattia Surricchio Sep 19 '20 at 15:30
  • 2
    $\begingroup$ To be honest, I believe that a lot of noise about oversampling comes from people who, sorry, don't know statistics and don't know how to think statistically, which in turn is mainly driven by the fact that much of "ML" has been appropriated by computer scientists, not the statistical community. $\endgroup$ – Stephan Kolassa Sep 19 '20 at 15:46
  • 2
    $\begingroup$ On Cross Validated, you are likely to get more statistical answers. I submit that our answers are better than oversampling. If you disagree, I would be very interested in your arguments, because I have never seen a serious one beyond "accuracy does weird things with unbalanced data", which statisticians have a (IMO) better answer to than oversampling. $\endgroup$ – Stephan Kolassa Sep 19 '20 at 15:46
  • 1
    $\begingroup$ @MattiaSurricchio F1 ignores true negatives and thus depends on which class you choose as positive. All of F1, accuracy, precision, etc depend on an assumption, often hidden and set to 0.5, about the probability cutoff from a model used to make a final class assignment. That probability cutoff should depend on the relative costs of false positives and false negatives. Much of what you read by computer-science/machine-learning people overlooks those issues. $\endgroup$ – EdM Sep 19 '20 at 16:02
  • 2
    $\begingroup$ @MattiaSurricchio here's an example of a good use for some type of F-score (there's a whole family, based on tradeoffs between precision and recall). In that case one is looking for objects, and there are no true-negative objects, just an object-free background. Some text-classification schemes are similar: you primarily want to identify some texts on a particular topic out of a large corpus and don't care much about off-topic texts in the corpus so long as you find enough on the topic. So "usefulness" is very application-dependent. $\endgroup$ – EdM Sep 19 '20 at 16:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.