0
$\begingroup$

Let's say my dataset contains 100k images. I have manually created 10 neural network models each with their own set of hyperparameters. I want to select the one which will perform the best on unseen data.

First I can divide my dataset into a training and test set, do a 5 fold cross validation on the training set to find the best performing model, then run the model on the test set to get a final estimate of its performance on unseen data. The downside is that I only have one test set to estimate my performance on and I can't train my model on the entire dataset.

Or I can do a nested cross validation which will allow me to train on my entire dataset, get a more accurate metric of its performance on unseen data, but with a downside that this could take weeks to train.

Or maybe I don't use cross validation at all and simply split my data into a train, validation, and test set. This will be fastest, but I lose the rigor of cross validation and I still can't train on my entire dataset.

Are all of these valid approaches? If I absolutely needed to train on all of my data and evaluate the final performance of my model am I forced to do nested cross validation?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.