Let's say my dataset contains 100k images. I have manually created 10 neural network models each with their own set of hyperparameters. I want to select the one which will perform the best on unseen data.

First I can divide my dataset into a training and test set, do a 5 fold cross validation on the training set to find the best performing model, then run the model on the test set to get a final estimate of its performance on unseen data. The downside is that I only have one test set to estimate my performance on and I can't train my model on the entire dataset.

Or I can do a nested cross validation which will allow me to train on my entire dataset, get a more accurate metric of its performance on unseen data, but with a downside that this could take weeks to train.

Or maybe I don't use cross validation at all and simply split my data into a train, validation, and test set. This will be fastest, but I lose the rigor of cross validation and I still can't train on my entire dataset.

Are all of these valid approaches? If I absolutely needed to train on all of my data and evaluate the final performance of my model am I forced to do nested cross validation?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.