I'm building a Random Forest model over an unbalaced 4 class dataset. So far I understood how to use oversampling and train my model. My doubt was about when to perform Oversampling.
I've already seen a lot of questions about oversampling before or after the train/test split, and I already know that the best way is to split into train/test before and then apply oversampling.
My doubt regards this second scenario (oversamplig after splitting).
Suppose that I have already splitted my dataset in train and test with a percentage of 80%-20% and I get my X_train, y_train, X_test, y_test data.
Now I'm going to perform (for example) cross validation over my X_train in order to estimate my validation error. For example (using Python) I could have something like:
from sklearn.model_selection import cross_val_score
from imblearn.pipeline import Pipeline, make_pipeline
from sklearn.ensemble import RandomForestClassifier
from imblearn.over_sampling import SMOTE
imba_pipeline = make_pipeline(SMOTE(sampling_strategy='auto', k_neighbors=10,random_state = SEED),
RandomForestClassifier(n_estimators=200, bootstrap=False, min_samples_leaf=2, min_samples_split=2, max_depth=14, random_state=SEED, class_weight='balanced',max_features = 'sqrt'))
scores=cross_val_score(imba_pipeline, X_train, y_train, scoring='accuracy', cv=10)
print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))
Now I'm happy with my crossvalidation score and I want to train my final model.
Should I retrain it over the X_train oversampled? So basically I'll do something like:
sm = SMOTE(sampling_strategy='auto', k_neighbors=10,random_state = SEED)
X_train_upsample, y_train_upsample = sm.fit_sample(X_train, y_train)
clf=RandomForestClassifier(n_estimators=200, bootstrap=False, min_samples_leaf=2, min_samples_split=2, max_depth=14, random_state=SEED, class_weight='balanced',max_features = 'sqrt')).fit(X_train_upsample, y_train_upsample)
Or is it a bad idea?
What if I performed crossvalidation on the already oversampled dataset? So, instead of oversampling each single fold, I have something like:
sm = SMOTE(sampling_strategy='auto', k_neighbors=10,random_state = SEED)
X_train_upsample, y_train_upsample = sm.fit_sample(X_train, y_train)
clf=RandomForestClassifier(n_estimators=200, bootstrap=False, min_samples_leaf=2, min_samples_split=2, max_depth=14, random_state=SEED, class_weight='balanced',max_features = 'sqrt'))
scores=cross_val_score(clf, X_train_upsample, y_train_upsample, scoring='accuracy', cv=10)
