Let's say my dataset contains 100k images. I have manually created 10 neural network models each with their own set of hyperparameters. I want to select the one which will perform the best on unseen data.
First I can divide my dataset into a training and test set, do a 5 fold cross validation on the training set to find the best performing model, then run the model on the test set to get a final estimate of its performance on unseen data. The downside is that I only have one test set to estimate my performance on and I can't train my model on the entire dataset.
Or I can do a nested cross validation which will allow me to train on my entire dataset, get a more accurate metric of its performance on unseen data, but with a downside that this could take weeks to train.
Or maybe I don't use cross validation at all and simply split my data into a train, validation, and test set. This will be fastest, but I lose the rigor of cross validation and I still can't train on my entire dataset.
Are all of these valid approaches? If I absolutely needed to train on all of my data and evaluate the final performance of my model am I forced to do nested cross validation?