Cross-validation for hyperparameter tuning

Question

I've read as many topics regarding hyperparameter tuning as I could, and I developed the following algorithm for hyperparameter tuning & final model building

1. Split the data in train set (80%) & test set (20%)
2. perform a k-fold cross-validation in the train set n times, changing the hyperparameters each time and choosing the ones that performed better in average on the validation sets.
3. build the neural networks that performed better in step 2 (let's say, the top 3) and train with the whole train set (80%)
4. Feed the NNs built in 3 with the test set and pick up the one that performed better.

I see no flaws in this process, however recently I've been reading about nested CV and I don't know when to use it instead of the steps proposed here.

Answer 1

Your flaw is in step 3 + 4:

• if you again want to select a model as optimal, you need to reserve yet another test set that is independent of steps 1 - 4.
• Moreover, the selection in step 4 is based on performance estimates that have only 1/4 of the test sample size compared to the selection in step 3 behind them. This makes that estimate more uncertain.

---

Nested cross validation means replacing the single train/test split (step 1 + evaluation of the one final model in step 4) by another, outer cross validation. I.e., a particular scheme to do those train/test splits repeatedly.