3
$\begingroup$

There is a theoretical and computational aspect to this question.

I was trying to use SMOTE to reduce class imbalance in a rather large dataset--about 8 million rows. The data has a binary outcome variable and 5 categorical variables. I was using the python imbalance-learn package, but the package basically used all 64 GB of my RAM and generally kept crashing with no result. Now that is an understandable outcome, since there adding dimensions to 8 million row matrices or computing the nearest neighbors, etc., is computationally expensive.

So I was trying to figure out strategies to handle the computation better. Since SMOTE, ADASYN, and other similar tools rely on nearest-neighbor matches, is there a way to breakdown the dataset into pieces, run the algorithm on them, and then reconstruct the total dataset? I have not seen any articles on something like this. I can think of a few different ways to do this, but I was not sure if there is any experimentation on something like this.

Improve this question
$\endgroup$
7
  • 6
    $\begingroup$ Why do you think unbalance is a problem? See stats.stackexchange.com/questions/283170/…, stats.stackexchange.com/questions/235808/…, stats.stackexchange.com/questions/6067/… $\endgroup$
    – kjetil b halvorsen ♦
    Commented Dec 22, 2019 at 13:43
  • $\begingroup$ I mean the imbalance problem is pretty obvious when I run the data. The output from training reads 99.999% accuracy while the False negative rate is 100% meaning that it gets every single True case wrong. By resampling this actually helped the problem. $\endgroup$
    – krishnab
    Commented Dec 22, 2019 at 16:55
  • 4
    $\begingroup$ Which method do you use for classification? Please read those linked posts better, and this one: stats.stackexchange.com/questions/404960/… Accuracy is not a proper scoring rule, and should not be used! $\endgroup$
    – kjetil b halvorsen ♦
    Commented Dec 22, 2019 at 23:57
  • 1
    $\begingroup$ @krishnab "meaning that it gets every single True case wrong. " that is not evidence of a class imbalance problem. For some learning tasks that is the optimal solution as the density of minority class patterns is never higher than that of majority class patterns anywhere in the input space. This seems to be a difficult problem to diagnose stats.stackexchange.com/questions/539638/… $\endgroup$
    – Dikran Marsupial
    Commented Aug 19, 2021 at 16:48
  • 1
    $\begingroup$ What’s wrong with getting $99.999\%$ accuracy? // I’d be interested in the ROCAUC of that model. With a gigantic class imbalance, it might be that the minority class never gets a predicted probability above the default cutoff threshold of $0.5$. ROCAUC is not a strictly proper scoring rule like log loss or Brier score, but it does not depend on a single threshold like accuracy or false negative rate. $\endgroup$
    – Dave
    Commented Dec 19, 2021 at 5:15

1 Answer 1

Reset to default
1
$\begingroup$

Regarding the computational question, I'd look into parallel computing. Maybe there's a way too split up the tasks and let every core run a part of the algorithm. Though the task combined with the size of the dataset is definitely not meant for a casual home PC.

Regarding the theoretical question (besides using resampling techniques like SMOTE, ROSE, ADASYN and many others) I'd look into cost-sensitive learning and switching to other performance metrics than accuracy which is definitely not what you want use for imbalanced data classification. Rather use the AUC, F1 score, Precision & Recall, etc.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.