Two separate repeated k-fold cross-validations vs. nested repeated k-fold cross-validation

Question

I have set up a code in R which does 2 different repeated cross-validations with different random-indices (two different set.seed) for random forest:

• the first random forest run with cross-validation tunes the parameter mtry using a gridsearch

• the second random forest run with cross-validation uses the optimized parameter mtry from the first run and produces the cross-Validation estimate of prediction error

Does my approach with the two separate cross-validation runs produce an error estimate which is also not optimistically biased? Would it be better to use nested cross-validation? Why would it be better?

Answer 1

 Does my approach with the two separate cross-validation runs produce an error estimate which is also not optimistically biased?

No.

 Would it be better to use nested cross-validation?

Yes

Why would it be better?

Because the 2nd CV of the 2 CVs after each other will use cases for testing that were already used for determining the optimal hyperparameters (in the 1st CV). Hyperparameter tuning is part of the training. Thus, the test cases of the 2nd CV are not independent.

Nested CV does test cases that were not used in hyperparameter tuning and are fully independent of the whole training process (given the splitting procedure correctly separates cases).