3
$\begingroup$

I recently had a (more experienced) coworker tell me that when you have a large enough sample, training error should be "good enough" to assess model performance. His point was that with a sufficiently large sample, resampling techniques become very computationally intensive while at the same time the training error rate becomes more accurate - and so it's not worth it to cross-validate.

For reference, we're working with traditional GLMs and GLMMs, with a dataset of about 20 million observations. A 10-folds cross-validation run for several different models takes a couple of hours to complete.

Is he right in that I'm wasting time with cross-validation?

$\endgroup$
2
$\begingroup$

Hard to say in general. It depends on your data. If you have enough predictors even a simple linear model can still overfit with 20 million observations.

In general, it's a valid point. The challenge can be to determine this empirically. One option would be to use random subsamples of about 10000 and show that the fitted models are more or less identical and use these models to predict the entire data and get the error.

The "predicting" the entire data might be a problem with a random factor, if it has enough levels. But in this case you have to think about whether you want to test predictive ability for new samples of the old random levels or whether you want to test the predictive ability for new samples with new random levels.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.