There is a theoretical and computational aspect to this question.

I was trying to use SMOTE to reduce class imbalance in a rather large dataset--about 8 million rows. The data has a binary outcome variable and 5 categorical variables. I was using the python imbalance-learn package, but the package basically used all 64 GB of my RAM and generally kept crashing with no result. Now that is an understandable outcome, since there adding dimensions to 8 million row matrices or computing the nearest neighbors, etc., is computationally expensive.

So I was trying to figure out strategies to handle the computation better. Since SMOTE, ADASYN, and other similar tools rely on nearest-neighbor matches, is there a way to breakdown the dataset into pieces, run the algorithm on them, and then reconstruct the total dataset? I have not seen any articles on something like this. I can think of a few different ways to do this, but I was not sure if there is any experimentation on something like this.


Regarding the computational question, I'd look into parallel computing. Maybe there's a way too split up the tasks and let every core run a part of the algorithm. Though the task combined with the size of the dataset is definitely not meant for a casual home PC.

Regarding the theoretical question (besides using resampling techniques like SMOTE, ROSE, ADASYN and many others) I'd look into cost-sensitive learning and switching to other performance metrics than accuracy which is definitely not what you want use for imbalanced data classification. Rather use the AUC, F1 score, Precision & Recall, etc.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.