Choosing a sample rate for GBM models

Question

I've created several GBM models to tune the parameters (trees, shrinkage and depth) to my data and the model performs well on the out-of-time sample. The data is credit card transactions (running into 100s of millions) so I sampled 1% of the good (non-event) and 100% of the bad.

However, when I increased the sample size to 3% of the good, there was a noticeable improvement in performance. My question is - how do I decide the optimal sampling rate, without running several iterations and deciding which one fits best? Is there a theory around this?

I have about 3 million total transactions (for the 1% sample), containing 380k bads and ~250 variables

P.S. Every iteration takes days to run, with run time increasing significantly as sample size is increased

Answer 1

"Learning When Training Data are Costly: The Effect of Class Distribution on Tree Induction"

See section 7.1. Test different undersampling rates using a small dataset.

Instead of using 3 million total examples, use the largest number you can where the model runs in a reasonable amount of time.

Vary the undersampling rate while using the same number of observations each time. The optimal undersampling rate on this small dataset should be a great estimate of the optimal undersampling rate to use with your full dataset.

Also, as a side note, I would recommend reviewing section three in the article. You might not be correcting for the bias in your final predictions. There are very simple methods to accomplish this. Otherwise as your class distributions approach the natural distribution, the bias will naturally decrease and it will seem like your predictions just keep getting better (even though you could get to the same result faster). So, either correct for it explicitly or use an error metric that adjusts for it.