So I've read around and seen this is a problem. I have a classification problem and 12 variables ... I'm working on getting more, but even if l get the number to 20-30 I feel like the problem will still persist because predicting all 0s still gives like 99.99% accuracy.
The problem: I have 250 1's and 4,000,000 0's in my data set. I need to get those 250 1's correct from an algorithm (in a test set) and dont mind throwing in a few thousand 0's.
What I'm doing right now: Sampling 1000 of the 4,000,000 0's and using a binary classification boosting tree model using all 250 1's.
What I want to know: What are some of the things to consider if I were to train say 200 different models using the same 250 1's and doing a random sample of 1000 from the 4,000,000 each time and then aggregating the the final decision over the entirety of the model 'ensemble' if you will?