Model selection procedure for large data sets using cross validation I have a question about model selection using cross validation. 
As far as I understood from many other replies related to model selection here, one should use nested cross validation in order to properly (1) select and then (2) assess a model. But almost all these questions were about relatively small data sets, so I assumed that this method is good for relatively small data sets. 
My question is what should one do in the case of a large data set ($\sim18$ millions observations, trying to select input variables out of $\sim10^2$ features), since training one model on such data set could be quite time consuming? Would it be fair to leave out test set for model assessment (after CV is done) and perform cross validation on rest of the data to select a model? 
 A: If we have $~18$ million observations, the first question I would attack is how much information are in these data, i.e., if we use a sample of data set, is it a bad idea?
An extreme case would be in these $18$ million observations say, 80% of them are redundant, (extreme example would be they are identical). So, we do not use all the data to do model building.
Learning curve is a tool (using cross validation) to tell us "how much data we need, and how complex the model we need.".  Here is an example.
How to know if a learning curve from SVM model suffers from bias or variance?
If you plot the learning curve for specific model, you may find 


*

*We are under fitting, which means the model is too simple, and data is very complex / contain a lot of information. Then, we can increase the model complexity

*We are over fitting, which means the model is too complex, and data is relative simple (say 18 million observations does not contain that much information as we expected), then use a sample of data would be sufficient.
I would suggest you use cross validation in "learning curve" way, which means you start with a sample of data, and increase the sample size, to observe training and testing loss to make further decision s on model selection.
