My data set is imbalanced - 5% of the target class represents fraudulent transactions, 95% of the target class represents legitimate transactions. I must use the whole data set, as the 95% of legitimate transactions are important for training. How can I use undersampling within algorithms such as rpart (decision tree), naive bayes, neural networks, SVM, etc. to create, run and evaluate using multiple splits of the data. For example: the number of legitimate transactions is equal to the number of fraudulent. So 5% and 5%. This is instead of the typical way of cutting down the data set to 50% legitimate, 50% fraud where you would lose 85% of the legitimate transactions. I cannot oversample by generating randomized data in this case.

undersampled data set


First off, 95%-5% is NOT an example of imbalanced dataset, I would consider downsampling if there was something of the order of 99.9%-0.1%. Most approaches should work just fine.

Edit: Seems like this answer has this covered already, have a look here: https://stats.stackexchange.com/a/133385/41367;

Now, if you have to rebalance, here are some options:

  1. Weight the infrequent class
  2. Stratified Sampling to downsize the frequent class
  3. Balanced subsampling to create multiple random balanced subsamples, then bootstrapping, ensembling (eg. Random Forests) etc.

From what I interpret, 3rd point should address your question about undersampling within the algorithm.

  • 4
    $\begingroup$ 95%-5% it's definitely not an example of balanced dataset either. You would get probably 100 percent of your samples predicted as majority class by any not cost sensitive classifier $\endgroup$ – rep_ho Oct 1 '16 at 13:15
  • $\begingroup$ Of course it isnt, if you were to take it literally. So, you are saying a coin toss with 95% probability of heads would be better than using any non-cost sensitive classifier? $\endgroup$ – tool.ish Oct 19 '16 at 18:43
  • $\begingroup$ @rep_ho The issue here is seeking hard classification without first fitting a probability model. If you do your modeling properly, 95%-5% is no problem for most probabilistic models. Cost optimization should be performed subsequent to a probabalistic model of your data. $\endgroup$ – Matthew Drury Apr 1 '17 at 17:39
  • $\begingroup$ That is true, but that was not mentioned in the question or the answer. If you are doing classifier, not a probabilistic model, most approaches will not work fine out of the box. But i agree that modeling probability should often prefered $\endgroup$ – rep_ho Apr 3 '17 at 10:58
  • $\begingroup$ Classifiers typically develop probability models implicitly so there is no need for an explicit model anyway. I'm curious, can you provide an example of a (out-of-the-box?) classifier that does not implicitly use a probability model? $\endgroup$ – tool.ish Apr 13 '17 at 12:39

Not the answer you're looking for? Browse other questions tagged or ask your own question.