1
$\begingroup$

I have about a 30% and 70% for class 0 (minority class) and class 1 (majority class). Since I do not have a lot of data, I am planning to oversample the minority class to balance out the classes to become a 50-50 split. I was wondering if oversampling should be done before or after splitting my data into train and test sets. I have generally seen it done before splitting in online examples, like this:

df_class0 = train[train.predict_var == 0]
df_class1 = train[train.predict_var == 1]
df_class1_over = df_class1.sample(len(df_class0), replace=True)
df_over = pd.concat([df_class0, df_class1_over], axis=0)

However, wouldn't that mean that the test data will likely have duplicated samples from the training set (because we have oversampled the training set)? This means that testing performance wouldn't necessarily be on new, unseen data. I am fine doing this, but I would like to know what is considered good practice. Thank you!

$\endgroup$
  • 2
    $\begingroup$ Thats a very weakly imbalanced dataset. You should only consider balancing if working with the data as is has failed. If you do decide to resample, model evaluation must be on unbalanced data, only the training set should be resampled. $\endgroup$ – Matthew Drury Jun 27 '18 at 14:51
  • 1
    $\begingroup$ Have a look at: stats.stackexchange.com/questions/6067/… $\endgroup$ – kjetil b halvorsen Jul 1 '18 at 19:07
1
$\begingroup$

In my opinion, using oversampling on your test set to balance classes is not a good practice. If you assume the distribution of class labels in your dataset is representative of what you are going to find in production/the real world, altering the test set is just going to give you a biased notion of how your algorithm is going to perform. The same can be said for other imbalance-fixing algorithms such as SMOTE which creates new synthetic samples. In my opinion, evaluating your algorithm on artificially generated samples is generally going to be less sound than using only real data. So I would only try to balance the training set, and probably select a test performance metric which is somehow informative in imbalanced situations (e.g., Precision/Recall rather than plain classification accuracy).

Also, I would recommend taking a look at more advanced algorithms for dataset balancing rather than over-sampling. The imbalanced-learn library might come in handy.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.