6
$\begingroup$

For different imbalanced data-sets which rare class' proportion differ from 30% (rare) to 5% (rare), what is the best way to define the Perc.Over and Perc.Under in SMOTE method of DMwR?

Currently I'm using this formula:

Over = ( (0.6 * COMMON_NO) - RARE_NO ) / RARE_NO

Under = (0.4 * COMMON_NO) / (RARE_NO * Over)

and finally transform it to percentage proportion:

Over_Perc = round(Over, 1) * 100

Under_Perc = round(Under, 1) * 100

and then feed it to SMOTE as following:

newData <- SMOTE(Label ~., df, perc.over=Over_Perc, perc.under=Under_Perc)

The thing is that, if the there is 30% rare data in the training set, performing classification without SMOTE gives a better performance than using SMOTE! While by having 5%-20% of rare data in the training set, using SMOTE, the classification algorithm gives a much more better performance,

p.s. I'm using Matthew's correlation and AUC for performance measurement.

$\endgroup$
2
  • $\begingroup$ You're talking about the SMOTE function in the R package DMwR? It would be better to be more explicit, including in your title. $\endgroup$
    – Glen_b
    Apr 6, 2013 at 5:54
  • 1
    $\begingroup$ @Glen_b The title edited to be more explicit, thanks. $\endgroup$
    – NULL
    Apr 6, 2013 at 8:54

1 Answer 1

5
$\begingroup$

I would be surprised if you did see improvement when you had 30% 'rare' data in the training set. 30% isn't really all that rare in the context of machine learning. What you could do is cross validate with various levels of synthetic data to determine what's giving you the best accuracy on your hold-out data (pretty standard approach to parameter tuning) and then go with that for your final model build. But I would be very surprised based upon personal experience if you saw significant gains in accuracy when you SMOTE past 20-25% positive class instances in your training set.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.