15
$\begingroup$

I am working on severely imbalanced data. In literature, several methods are used to re-balance the data using re-sampling (over- or under-sampling). Two good approaches are:

  • SMOTE: Synthetic Minority Over-sampling TEchnique (SMOTE)

  • ADASYN: Adaptive Synthetic Sampling Approach for Imbalanced Learning (ADASYN)

I have implemented ADASYN because its adaptive nature and ease to extension to multi-class problems.

My question is how to test the oversampled data produced by ADASYN (or any other oversampling methods). It is not clear in the mentioned two paper how they performed their experiments. There are two scenarios:

1- Oversample the whole dataset, then split it to training and testing sets (or cross validation).

2- After splitting the original dataset, perform oversampling on the training set only and test on the original data test set (could be performed with cross validation).

In the first case the results are much better than without oversampling, but I am concerned if there is overfitting. While in the second case the results are slightly better than without oversampling and much worse than the first case. But the concern with the second case is if all minority class samples goes to the testing set, then no benefit will be achieved with oversampling.

I am not sure if there are any other settings to test such data.

$\endgroup$
14
$\begingroup$

A few comments:

The option (1) is a very bad idea. Copies of the same point may end up in both the training and test sets. This allows the classifier to cheat, because when trying to make predictions on the test set the classifier will already have seen identical points in the train set. The whole point of having a test set and a train set is that the test set should be independent of the train set.

The option (2) is honest. If you don't have enough data, you could try using $k$-fold cross validation. For example, you could divide your data into 10 folds. Then, for each fold individually, use that fold as the test set and the remaining 9 folds as a train set. You can then average training accuracy over the 10 runs. The point of this method is that since only 1/10 of your data is in the test set, it is unlikely that all your minority class samples end up in the test set.

$\endgroup$
  • $\begingroup$ Thanks Stefan for the reply. But I want to clarify a point: The methods I mentioned create "synthetic" instances of the data, not exactly the same as the original one. So it is not exactly as if I am testing on the same training data. But, still I am not sure if option 1 is a valid approach. $\endgroup$ – Fares May 28 '13 at 9:59
  • $\begingroup$ Ok I see! The problem with (1) still holds, though, even if you create "synthetic" data, because the synthetic data is usually very similar to the original data it is mimicking. $\endgroup$ – Stefan Wager May 28 '13 at 16:23
  • $\begingroup$ @StefanWager what about the distribution of testing set? Should it be the same as the original data set? $\endgroup$ – wannik Dec 14 '14 at 20:04
  • 1
    $\begingroup$ any reference of articles to support this claim? $\endgroup$ – girl101 Mar 25 at 6:08
4
$\begingroup$

The second (2) option is the right way of doing it. The synthetic samples you create with the oversampling techniques are not real examples but rather synthetic. These are not valid for testing purposes while they still ok for training. They are intended to modify the behavior of the classifier without modifying the algorithm.

$\endgroup$

protected by kjetil b halvorsen Apr 24 '18 at 17:02

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.