30
$\begingroup$

I am working on severely imbalanced data. In literature, several methods are used to re-balance the data using re-sampling (over- or under-sampling). Two good approaches are:

  • SMOTE: Synthetic Minority Over-sampling TEchnique (SMOTE)

  • ADASYN: Adaptive Synthetic Sampling Approach for Imbalanced Learning (ADASYN)

I have implemented ADASYN because its adaptive nature and ease to extension to multi-class problems.

My question is how to test the oversampled data produced by ADASYN (or any other oversampling methods). It is not clear in the mentioned two paper how they performed their experiments. There are two scenarios:

1- Oversample the whole dataset, then split it to training and testing sets (or cross validation).

2- After splitting the original dataset, perform oversampling on the training set only and test on the original data test set (could be performed with cross validation).

In the first case the results are much better than without oversampling, but I am concerned if there is overfitting. While in the second case the results are slightly better than without oversampling and much worse than the first case. But the concern with the second case is if all minority class samples goes to the testing set, then no benefit will be achieved with oversampling.

I am not sure if there are any other settings to test such data.

Improve this question
$\endgroup$
0

3 Answers 3

Reset to default
25
$\begingroup$

A few comments:

The option (1) is a very bad idea. Copies of the same point may end up in both the training and test sets. This allows the classifier to cheat, because when trying to make predictions on the test set the classifier will already have seen identical points in the train set. The whole point of having a test set and a train set is that the test set should be independent of the train set.

The option (2) is honest. If you don't have enough data, you could try using $k$-fold cross validation. For example, you could divide your data into 10 folds. Then, for each fold individually, use that fold as the test set and the remaining 9 folds as a train set. You can then average training accuracy over the 10 runs. The point of this method is that since only 1/10 of your data is in the test set, it is unlikely that all your minority class samples end up in the test set.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thanks Stefan for the reply. But I want to clarify a point: The methods I mentioned create "synthetic" instances of the data, not exactly the same as the original one. So it is not exactly as if I am testing on the same training data. But, still I am not sure if option 1 is a valid approach. $\endgroup$
    – Fares
    Commented May 28, 2013 at 9:59
  • $\begingroup$ Ok I see! The problem with (1) still holds, though, even if you create "synthetic" data, because the synthetic data is usually very similar to the original data it is mimicking. $\endgroup$
    – Stefan Wager
    Commented May 28, 2013 at 16:23
  • $\begingroup$ @StefanWager what about the distribution of testing set? Should it be the same as the original data set? $\endgroup$
    – wannik
    Commented Dec 14, 2014 at 20:04
  • 3
    $\begingroup$ any reference of articles to support this claim? $\endgroup$
    – girl101
    Commented Mar 25, 2019 at 6:08
  • $\begingroup$ @stefan’s approach is correct you never augment or synthesize your validation set because thats your proof (test) of how your model is learning, you need original data to do so $\endgroup$
    – M090009
    Commented Nov 19, 2019 at 10:25
14
$\begingroup$

The second (2) option is the right way of doing it. The synthetic samples you create with the oversampling techniques are not real examples but rather synthetic. These are not valid for testing purposes while they still ok for training. They are intended to modify the behavior of the classifier without modifying the algorithm.

Improve this answer
$\endgroup$
0
$\begingroup$

Do not do either one of these two approaches. Unbalanced data is not a problem, and oversampling will not solve a non-problem. Are unbalanced datasets problematic, and (how) does oversampling (purport to) help?

This Meta.CV thread contains a curated list of useful links on imbalanced data.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.