0
$\begingroup$

I've read as many topics regarding hyperparameter tuning as I could, and I developed the following algorithm for hyperparameter tuning & final model building

  1. Split the data in train set (80%) & test set (20%)
  2. perform a k-fold cross-validation in the train set n times, changing the hyperparameters each time and choosing the ones that performed better in average on the validation sets.
  3. build the neural networks that performed better in step 2 (let's say, the top 3) and train with the whole train set (80%)
  4. Feed the NNs built in 3 with the test set and pick up the one that performed better.

I see no flaws in this process, however recently I've been reading about nested CV and I don't know when to use it instead of the steps proposed here.

$\endgroup$

1 Answer 1

1
$\begingroup$

Your flaw is in step 3 + 4:

  • if you again want to select a model as optimal, you need to reserve yet another test set that is independent of steps 1 - 4.
  • Moreover, the selection in step 4 is based on performance estimates that have only 1/4 of the test sample size compared to the selection in step 3 behind them. This makes that estimate more uncertain.

Nested cross validation means replacing the single train/test split (step 1 + evaluation of the one final model in step 4) by another, outer cross validation. I.e., a particular scheme to do those train/test splits repeatedly.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.