5
$\begingroup$

I'm using Random Forest in the CARET package to tag a binary outcome with 1/10 ratio, thus I need to balance the dataset.

I know two ways:

  1. Use SMOTE as a stand-alone function and then pass it to the training.

  2. Use sampling='smote' inside CARET's training

As far as I understand, the first approach should be better, for it uses the whole data set to synthesize new samples (I know it uses only the 5 nearest neighbors by default, but still have more data points to choose from) while the second method only uses the data points available in each partition of the CV.

However, are there any benefits in balancing inside the CV?

$\endgroup$
1

2 Answers 2

8
$\begingroup$

The second method should be preferred for exactly the reason that you gave to justify the first. The first method uses the whole data set to synthesize new samples. Cross validation is excluding points from training to give an accurate assessment of the error rate on new data. If you use SMOTE first, information from the excluded points will be leaked into the training data and will taint the XV testing.

$\endgroup$
1
  • 2
    $\begingroup$ I see, basically I synthesise more "never seen before" data points. $\endgroup$ May 17, 2018 at 7:37
2
$\begingroup$

Method 1 should not be used as it leaks information from the test partition into the training set in each fold of the cross-validation. This is because a synthetic example may lie between a real training pattern and a real test pattern or between two real test patterns. Consider a synthetic example generated by random chance very close to the real test pattern ending up in the training set.

The way to look at it is that cross-validation is a method of evaluating the performance of a procedure for fitting a model, rather than of the model itself. So the whole procedure must be implemented independently, in full, in each fold of the cross-validation. So if SMOTE is part of the model-fitting procedure, it needs to be done separately in each fold.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.