1
$\begingroup$

Was reading an article on imbalanced datasets where the event occurs and look at balancing the dataset. In that article, the event records were 2% of the total records. The author of the blog suggested to do Random Over Sampling and increase the total number of event records to 10% of the total records.

Raw Data Stats: Total Observations = 1000 Fraudulent Observations = 20 Non Fraudulent Observations = 980 Event Rate= 2 %

My understanding was that we should make the event and the non-event records 50% each to balance the dataset. Wanted to understand the correct approach.

$\endgroup$
  • 1
    $\begingroup$ The correct approach is to not balance the data at all until working with the raw data has failed. A 2% imbalance is well within the realm where working with raw data will often work out fine. $\endgroup$ – Matthew Drury Jul 16 '18 at 14:44
  • $\begingroup$ @Matthew Drury I think I have not articulated properly - The data set is highly imbalanced. e.g. Total Observations = 1000;Fraudulent Observations = 20;Non Fraudulent Observations = 980;Event Rate= 2 %. Shall add this to the main question. Kindly let me know your thoughts. $\endgroup$ – Bonson Jul 17 '18 at 0:49
1
$\begingroup$

My answer is stated in terms of undersampling (sample the common event to get balance) whereas the OP is asking about oversampling (sampling w/replacement or upweighting the rare events). Both approaches will get you to a balance of response types and the mathematical arguments are similar (why I'm keeping my post up rather than deleting) but the implementations differ (which thing you sample and how you sample-or upweight).

Everything in my answer about downsampling (mutatis mutandis) should also be true for oversampling.

It really depends on which model you're working with. For example, with logistic regression it is very common approach to sample from the common event to induce more balance. However, this approach is very sensitive to correct specification of model and the feature set. The reason this is done is clearly explained in an article by Gary King and one of his students (now a prof at UCSD). There is also a really nice article by Will Fithian and Trevor Hastie about iterating this approach.

Frank Harrell has commented on another related post and his opinion is not to do the sampling. I disagree with his opinion. I've used the case-control approach in applied work several times to much success. I've also studied how and why it works or doesn't. If you have lots of data it really shouldn't matter but if you samples are expensive to collect and the sample size isn't big enough for traditional asymptotics to kick in, I'd use the down sampling approach.

For Support Vector Machines (SVMs), this isn't necessary as far as I can tell, and I'm unaware if this is done for SVMs. I don't think it would hurt all that much though.

It is unclear to me if this is an issue with Neural nets though. Maybe someone else will chime in on that aspect in a different comment. If not and I get some time to investigate I'll come back and update my answer, but I wouldn't wait around for me to do that, might be awhile.

$\endgroup$
  • 1
    $\begingroup$ The OP is talking about oversampling, not downsampling. $\endgroup$ – Bryan Krause Aug 14 '18 at 22:25
  • 1
    $\begingroup$ @Bryan Krause, I expect the conclusions should be the same. Are you suggesting they should not be the same? If so, please tell me why, that is not obvious to me. $\endgroup$ – Lucas Roberts Aug 14 '18 at 22:27
  • $\begingroup$ I suspect some of the general pros and cons will be the same, but the algorithms are certainly different and the approach is a bit different conceptually, though they can also be mixed. Personally I feel a bit more discomfort with the idea of oversampling but I don't have an empirical reason for that discomfort. I still think bringing undersampling into the conversation is relevant, just doesn't directly address OP's question. $\endgroup$ – Bryan Krause Aug 14 '18 at 23:20
  • 1
    $\begingroup$ @Bryan Krause, fair enough, I'll edit my answer to make it clear I'm talking about undersampling techniques rather than oversampling, that the algorithms are different (I'll make clear generally how they differ) and that the points I'm making should hold for oversampling too. Unless of course you think something I've stated in the answer won't hold for over-sampling. $\endgroup$ – Lucas Roberts Aug 14 '18 at 23:47
  • $\begingroup$ @LucasRoberts thank you for your response. Did under and over sampling for a dataset. In Under Sampling Train Data Label Distribution: False. 435 True. 435 Confusion Matrix: [[259 27] [ 9 39]] Prec_Score_false: 0.96 Prec_Score_true: 0.59 Recall_Score_false: 0.90 Recall_Score_true: 0.81 --------- In Over Sampling: Train Data Label Distribution: True. 2564 False. 2564 Confusion Matrix: [[262 24] [ 13 35]] Prec_Score_false: 0.95 Prec_Score_true: 0.59 Recall_Score_false: 0.92 Recall_Score_true: 0.73 $\endgroup$ – Bonson Sep 6 '18 at 23:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.