1
$\begingroup$

Was reading an article on imbalanced datasets where the event occurs and look at balancing the dataset. In that article, the event records were 2% of the total records. The author of the blog suggested to do Random Over Sampling and increase the total number of event records to 10% of the total records.

Raw Data Stats: Total Observations = 1000 Fraudulent Observations = 20 Non Fraudulent Observations = 980 Event Rate= 2 %

My understanding was that we should make the event and the non-event records 50% each to balance the dataset. Wanted to understand the correct approach.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ The correct approach is to not balance the data at all until working with the raw data has failed. A 2% imbalance is well within the realm where working with raw data will often work out fine. $\endgroup$
    – Matthew Drury
    Jul 16, 2018 at 14:44
  • $\begingroup$ @Matthew Drury I think I have not articulated properly - The data set is highly imbalanced. e.g. Total Observations = 1000;Fraudulent Observations = 20;Non Fraudulent Observations = 980;Event Rate= 2 %. Shall add this to the main question. Kindly let me know your thoughts. $\endgroup$
    – Bonson
    Jul 17, 2018 at 0:49

1 Answer 1

Reset to default
1
$\begingroup$

My answer is stated in terms of undersampling (sample the common event to get balance) whereas the OP is asking about oversampling (sampling w/replacement or upweighting the rare events). Both approaches will get you to a balance of response types and the mathematical arguments are similar (why I'm keeping my post up rather than deleting) but the implementations differ (which thing you sample and how you sample-or upweight).

Everything in my answer about downsampling (mutatis mutandis) should also be true for oversampling.

It really depends on which model you're working with. For example, with logistic regression it is very common approach to sample from the common event to induce more balance. However, this approach is very sensitive to correct specification of model and the feature set. The reason this is done is clearly explained in an article by Gary King and one of his students (now a prof at UCSD). There is also a really nice article by Will Fithian and Trevor Hastie about iterating this approach.

Frank Harrell has commented on another related post and his opinion is not to do the sampling. I disagree with his opinion. I've used the case-control approach in applied work several times to much success. I've also studied how and why it works or doesn't. If you have lots of data it really shouldn't matter but if you samples are expensive to collect and the sample size isn't big enough for traditional asymptotics to kick in, I'd use the down sampling approach.

For Support Vector Machines (SVMs), this isn't necessary as far as I can tell, and I'm unaware if this is done for SVMs. I don't think it would hurt all that much though.

It is unclear to me if this is an issue with Neural nets though. Maybe someone else will chime in on that aspect in a different comment. If not and I get some time to investigate I'll come back and update my answer, but I wouldn't wait around for me to do that, might be awhile.

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ The OP is talking about oversampling, not downsampling. $\endgroup$
    – Bryan Krause
    Aug 14, 2018 at 22:25
  • 1
    $\begingroup$ @Bryan Krause, I expect the conclusions should be the same. Are you suggesting they should not be the same? If so, please tell me why, that is not obvious to me. $\endgroup$
    – Lucas Roberts
    Aug 14, 2018 at 22:27
  • $\begingroup$ I suspect some of the general pros and cons will be the same, but the algorithms are certainly different and the approach is a bit different conceptually, though they can also be mixed. Personally I feel a bit more discomfort with the idea of oversampling but I don't have an empirical reason for that discomfort. I still think bringing undersampling into the conversation is relevant, just doesn't directly address OP's question. $\endgroup$
    – Bryan Krause
    Aug 14, 2018 at 23:20
  • 1
    $\begingroup$ @Bryan Krause, fair enough, I'll edit my answer to make it clear I'm talking about undersampling techniques rather than oversampling, that the algorithms are different (I'll make clear generally how they differ) and that the points I'm making should hold for oversampling too. Unless of course you think something I've stated in the answer won't hold for over-sampling. $\endgroup$
    – Lucas Roberts
    Aug 14, 2018 at 23:47
  • $\begingroup$ @LucasRoberts thank you for your response. Did under and over sampling for a dataset. In Under Sampling Train Data Label Distribution: False. 435 True. 435 Confusion Matrix: [[259 27] [ 9 39]] Prec_Score_false: 0.96 Prec_Score_true: 0.59 Recall_Score_false: 0.90 Recall_Score_true: 0.81 --------- In Over Sampling: Train Data Label Distribution: True. 2564 False. 2564 Confusion Matrix: [[262 24] [ 13 35]] Prec_Score_false: 0.95 Prec_Score_true: 0.59 Recall_Score_false: 0.92 Recall_Score_true: 0.73 $\endgroup$
    – Bonson
    Sep 6, 2018 at 23:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.