I have a dataset with around 3.4 million records. I am predicting a binary target variable. The distribution of the target variable is 10%. I am using 17 features for prediction and am comparing performance using stratified 5 fold cross-validation.
The main way I measure performance is by looking at ROC AUC. In my cross validation i get an average AUC of around .65. When I limit my training portion to around 10% of the data (300,000 records) and test it on the remaining 90 % i get an AUC of .73 and substantial increases in lift at lower thresholds.
My question is why could this be the case? Is using too much data for training overfitting?
Additional note. I used python sci-kit learn inplentation using the default parameters with 300 trees and a min_samples_split of .0005.
Any recommendations or reasons why I am getting these score discrepancies?
I think have solved this issue. I wanted to note what I was doing wrong.
The data I am using is geographical data and was created by concatenated several different datasets. The datasets were concatenated in a way so certain geographical segments were put together and it was not in random order.
I used sci kit learns implementation of stratified k fold to create the splits in my cross validation.
By default this does not shuffle the data before creating the splits and does the following (direct quote from documentation)
"Preserve order dependencies in the dataset ordering, when shuffle=False: all samples from class k in some test set were contiguous in y, or separated in y by samples from classes other than k."
I specifically do not want to do this because of the way my training dataset is created, i would prefer my data set to be shuffled randomly because preserving order in this way can create biases for specific geographical segments.
When I was experimenting with smaller datasets I was using a simple pandas df.sample. This selects random sample of the data and does not care about order. This is why when i trained on this data i saw better results.
The way I fixed the issue with the cross validation is simply making sure the data gets shuffled before I make my splits. This is done by passing the shuffle = True parameter into sci-kit learns stratified k fold.