I am working on severely imbalanced data. In literature, several methods are used to re-balance the data using re-sampling (over- or under-sampling). Two good approaches are:
SMOTE: Synthetic Minority Over-sampling TEchnique (SMOTE)
ADASYN: Adaptive Synthetic Sampling Approach for Imbalanced Learning (ADASYN)
I have implemented ADASYN because its adaptive nature and ease to extension to multi-class problems.
My question is how to test the oversampled data produced by ADASYN (or any other oversampling methods). It is not clear in the mentioned two paper how they performed their experiments. There are two scenarios:
1- Oversample the whole dataset, then split it to training and testing sets (or cross validation).
2- After splitting the original dataset, perform oversampling on the training set only and test on the original data test set (could be performed with cross validation).
In the first case the results are much better than without oversampling, but I am concerned if there is overfitting. While in the second case the results are slightly better than without oversampling and much worse than the first case. But the concern with the second case is if all minority class samples goes to the testing set, then no benefit will be achieved with oversampling.
I am not sure if there are any other settings to test such data.