4
$\begingroup$

We've ran into a problem while training a classifier on an unbalanced data set.

The response is binary with 0 indicating 'non defaulter' and 1 indicating 'defaulter' (it's a credit scoring task).

The defaulters only account for 0.47 % (233 observations of 47k-ish). We used the SMOTE algorithm to over sample the minority class and thus balancing the data set. This is a known method highly encouraged in credit-scoring applications, or in any other classifying situations where the distribution in the response is skewed.

We tested different volumes of SMOTE-oversampling, but finally settled with a data set where the ratio of defaulters is about 40 %, which means the SMOTE algorithm produced about 26567 artificially created observations. This set now holds the TRUE defaulters, the non defaulters and the artificially created observations which of course are all labeled as defaulters.

After partitioning the data we've trained different kinds of classifiers and compared their results using the "Holdout method" (EG using a test set). The most successful classifiers was a neural network with one hidden layer (holding 30 neurons), feed forward structure, weight "optimizer" used was back propagation, and the actiovation function was set to hyperbolic. We also used an ensembler since we noticed that boosting (10 loops) increased the prediction accuracy.

The model produces very good results overall. For example, the True Positive Rate is about 93 %, and the True Negative Rate is even higher. The area under the classifiers ROC-curve is well over 0.9.

We were strutting around like glorified roosters after creating such an über model - until we decided to label the 233 TRUE defaulters and do a follow up on how they were classified. To our horror the model only classified about 60 % av them to the defaulting class.

Our guess is that the SMOTE algorithm might have went a little bit crazy and overlapped into the non-defaulting group a bit too much when creating the artificial defaulters.

Is there a way to prevent this from happening? Is undersampling the majority class of non defaulters and combining this with a SMOTE-oversampler of say 20 % a good approach? Why and why not?

We ran into something called Tomek links which seems to "inverse" the risky effects of SMOTEing a bit.

$\endgroup$
  • $\begingroup$ I do not have an answer. But I've got the same problem: stats.stackexchange.com/questions/61622/… . If you happen to have a solution for it, I'd also be grateful. $\endgroup$ – Olivier_s_j Jun 13 '13 at 14:25
  • 1
    $\begingroup$ We presented a solution based on Chawla's approach which includes combining SMOTE with random under-sampling. Also, we scaled up the true defaulters 10x, simply by performing random over sampling of the minority class. Boosting might be worth looking in to as well. $\endgroup$ – Eric Paulsson Jun 16 '13 at 21:50
  • $\begingroup$ You could try RUSBoost or other cost sensitive ML techniques for the same. This does not require under or over sampling. $\endgroup$ – prashanth Jun 30 '16 at 11:19
  • $\begingroup$ What kind of model are you using? Only non-probabilistic models need balancing, so probabilistic models should be preferred. $\endgroup$ – Matthew Drury Feb 3 '17 at 22:13
6
$\begingroup$

It seems like you are oversampling (i.e. generating synthetic data instances) before splitting training and testing data. This causes over-fitting and hence your optimistic initial results. As pointed here, you should consider applying oversampling after splitting your data.

$\endgroup$
  • 2
    $\begingroup$ I second this. If you are using Python, you can pipeline your SMOTE step so that it will be automatically applied separately to the training set after splitting the training and test sets. Also, the PR AUC is a more informative metric than the ROC AUC. XGBoost and RandomForestClassifier can handle unbalanced data when the proper argument is passed to them (scale_pos_weight = ... for the former and class_weight = 'balanced' for the latter). $\endgroup$ – darXider Apr 7 '17 at 16:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.