Process for oversampling data for imbalanced binary classe

Question

I have about a 30% and 70% for class 0 (minority class) and class 1 (majority class). Since I do not have a lot of data, I am planning to oversample the minority class to balance out the classes to become a 50-50 split. I was wondering if oversampling should be done before or after splitting my data into train and test sets. I have generally seen it done before splitting in online examples, like this:

df_class0 = train[train.predict_var == 0]
df_class1 = train[train.predict_var == 1]
df_class1_over = df_class1.sample(len(df_class0), replace=True)
df_over = pd.concat([df_class0, df_class1_over], axis=0)

However, wouldn't that mean that the test data will likely have duplicated samples from the training set (because we have oversampled the training set)? This means that testing performance wouldn't necessarily be on new, unseen data. I am fine doing this, but I would like to know what is considered good practice. Thank you!

Answer 1

In my opinion, using oversampling on your test set to balance classes is not a good practice. If you assume the distribution of class labels in your dataset is representative of what you are going to find in production/the real world, altering the test set is just going to give you a biased notion of how your algorithm is going to perform. The same can be said for other imbalance-fixing algorithms such as SMOTE which creates new synthetic samples. In my opinion, evaluating your algorithm on artificially generated samples is generally going to be less sound than using only real data. So I would only try to balance the training set, and probably select a test performance metric which is somehow informative in imbalanced situations (e.g., Precision/Recall rather than plain classification accuracy).

Also, I would recommend taking a look at more advanced algorithms for dataset balancing rather than over-sampling. The imbalanced-learn library might come in handy.