Machine learning strategy for imbalanced data with high number of examples I am working on a classification problem, with unbalanced classes :
Number of positive examples: ~200k;  Number of negative example is ~230 Millions examples.
The only two requirements that I have is: Using AUC for evaluation and the evaluation should be at natural rate i.e. 200k/230M = 0.0008.
My question here is, knowing that using all that data is quite impossible because of performance constraints (currently the limit is around 6M sample of the data), what would be the strategy to train, cv and test a model ?
Two propositions came up, we can't decide which one is the best practice in this case:

*

*Train on 6M neg + 190k pos, Crossvalidation using natural rate, Test using natural rate

*Train on 6M neg + 5k pos (natural rate), CV and Test are all in natural rate

 A: I disagree with user2974951's answer. You should aim at probabilistic classifications that are well-calibrated and sharp. Oversampling the minority class will bias your predictions, so don't do it. Use a representative sample of your initial data for training your model.
Once you have your probabilistic classifications, you can calculate your AUC. In addition, you can tweak your cutoff threshold (which I am not too keen on) based on your training sample, until you get a "positive" hard classification rate of 0.008, as the client requires. (Which doesn't make much sense to me, per the same reasoning.)
A: If you must follow the requirements, then the first option would be better for the simple reason that you have more data for the minority class, so you should be able to capture more of the variability in this class. As opposed to the second option where you would first have to (find a way to) select a small subset of all the data (and throw the other data out?, this seems unnecessarily wasteful).
Edit: some more clarification. The first option would see you select a sample of your data (although much bigger than the second option) and the perform CV with folds, ensuring that each fold is trained with the same ratio of positives and negatives, as your original sample. In this way you would preserve the required "natural rate" while at the same time having more data available, so I don't see how this is a worse option than number 2.
I would, however, suggest a third option, one that would require no removing of data. Keep all your data as is, but then perform a generalization of CV, whereby you train your desired model multiple times with subsamples of your data, while keeping the ratio intact (stratified sampling). So each iteration you select a new subsample, getting new data into your training sample. This way, given enough repetitions, you should get most of your data through your model, and hence get a much better look at the performance of the model. The best cutoff for AUC would then be selected from the folds, where the natural rate was in effect.
This is similar to repeated CV, where you repeat CV multiple times, each time with new folds, so as to try and randomize the training and test data as much as possible, and get a more accurate result.
