1
$\begingroup$

Consider a situation, where there are two unbalanced classes (n1 < n2). Some standard statistical methods advise to use SMOTE (or similar) oversampling methods to balance classes and train a classifier on balanced classes.

Here is my question: how to assess properly a classification error of the trained classifier (or any other metrics, f1, etc)? Remember, the classifier is trained on the augmented dataset (balanced).

Does cross validation on the augmented dataset make sense? I was told, that assessing a performance of the classifier (which is trained on balanced data) should be done only on the original unbalanced dataset.

UPD Here is a nice paper addressing exactly the same problem of CV with SMOTE

|cite|improve this question
$\endgroup$
1
$\begingroup$

Cross validation on the SMOTE-augmented dataset doesn't make sense, because:

  1. it would contaminate the training sets with the test sets.
  2. it changes the test sets.

You should apply SMOTE only to the training set, i.e. without using the test set. If you want to report cross-validation results, at each iteration of the cross-validation, you should apply SMOTE to the training set.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ thanks for amazing answer! But I don't quite understand, how SMOTE may affect the training set (contaminate/changes). Can you elaborate on this? $\endgroup$ – Arnold Klein Oct 20 '16 at 19:30
  • $\begingroup$ The caret documentation on dealing with imbalanced classes might be helpful to look at for some clarification: topepo.github.io/caret/subsampling-for-class-imbalances.html $\endgroup$ – dmartin Oct 20 '16 at 20:14
  • $\begingroup$ @dmartin, Does the problem of "contamination" of the test set in CV come from the fact, that oversampling techniques may replicate exactly the same data and in CV process, the same data points (the original, and the replicated one) may appear in both, training and the test sets simultaneously? $\endgroup$ – Arnold Klein Oct 24 '16 at 11:00
  • 1
    $\begingroup$ @ArnoldKlein correct, either exact or approximate replicate $\endgroup$ – Franck Dernoncourt Oct 24 '16 at 13:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.