I have a question about model selection using cross validation.

As far as I understood from many other replies related to model selection here, one should use nested cross validation in order to properly (1) select and then (2) assess a model. But almost all these questions were about relatively small data sets, so I assumed that this method is good for relatively small data sets.

My question is what should one do in the case of a large data set ($\sim18$ millions observations, trying to select input variables out of $\sim10^2$ features), since training one model on such data set could be quite time consuming? Would it be fair to leave out test set for model assessment (after CV is done) and perform cross validation on rest of the data to select a model?


If we have $~18$ million observations, the first question I would attack is how much information are in these data, i.e., if we use a sample of data set, is it a bad idea?

An extreme case would be in these $18$ million observations say, 80% of them are redundant, (extreme example would be they are identical). So, we do not use all the data to do model building.

Learning curve is a tool (using cross validation) to tell us "how much data we need, and how complex the model we need.". Here is an example. How to know if a learning curve from SVM model suffers from bias or variance?

If you plot the learning curve for specific model, you may find

  • We are under fitting, which means the model is too simple, and data is very complex / contain a lot of information. Then, we can increase the model complexity

  • We are over fitting, which means the model is too complex, and data is relative simple (say 18 million observations does not contain that much information as we expected), then use a sample of data would be sufficient.

I would suggest you use cross validation in "learning curve" way, which means you start with a sample of data, and increase the sample size, to observe training and testing loss to make further decision s on model selection.

  • $\begingroup$ Thank you very much @hxd1011. This is a great point. I would like to try out this approach. I just have several questions about it. I apologize in advance if the questions are stupid, I just probably need to read more about learning curve. I would greatly appreciate if you could answer or point me to the resources where I could find relevant information. My questions are: (1) Could you please give me an advise on how to increase the sample if the data I need to use is very skewed? Should I use some sort of stratification when I add samples or just add them randomly to the training set? $\endgroup$ – Ira Z. Feb 16 '17 at 16:21
  • $\begingroup$ (2) Regarding CV, should I apply such procedure as in (1) to each fold? I mean increasing the size of folds (I assume by taking future samples from some kind of poll?). (3) If I want to conduct model selection, do I use this procedure for each hyper-parameters/features/etc. combination? Thank you very much for your help! $\endgroup$ – Ira Z. Feb 16 '17 at 16:21
  • $\begingroup$ @IraZ. check this video. youtube.com/watch?v=e3edL-_fUTo $\endgroup$ – Haitao Du Feb 16 '17 at 16:28
  • $\begingroup$ @IraZ. no problem. You are thinking too much on how to sample the data. If you have 18 million data, how do you know it is skewed? we can start with a small sample and observe the trend and increase it. During the process you will know better about the data. $\endgroup$ – Haitao Du Feb 16 '17 at 16:38
  • $\begingroup$ Thanks again @hxd1011! I just have to ask one more question. How do I compare different models using this method? I have a big set of features from which I need to select the relevant ones, and for each of the feature set I would then need to find the optimal model parameters (number of neurons in the hidden layer of feedforward neural network in my case). Is there some conventional way to do it using the learning curve? $\endgroup$ – Ira Z. Feb 22 '17 at 12:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.