# Running SMOTE on very large class imbalanced datasets - batched or subsampled implementations

There is a theoretical and computational aspect to this question.

I was trying to use SMOTE to reduce class imbalance in a rather large dataset--about 8 million rows. The data has a binary outcome variable and 5 categorical variables. I was using the python imbalance-learn package, but the package basically used all 64 GB of my RAM and generally kept crashing with no result. Now that is an understandable outcome, since there adding dimensions to 8 million row matrices or computing the nearest neighbors, etc., is computationally expensive.

So I was trying to figure out strategies to handle the computation better. Since SMOTE, ADASYN, and other similar tools rely on nearest-neighbor matches, is there a way to breakdown the dataset into pieces, run the algorithm on them, and then reconstruct the total dataset? I have not seen any articles on something like this. I can think of a few different ways to do this, but I was not sure if there is any experimentation on something like this.