This question already has an answer here:
- Use of nested cross-validation 2 answers
Imagine I have a classifier with some parameter $a$. I want to perform parameter optimisation and model selection. Imagine that my grid for the values of $a$ has three values 1,5 and 10 and I want perform 10-fold cross-validation as follows:
For each iteration, the dataset is partitioned into a 9:1 ratio of training and testdata, whereby at each iteration the two consists from other sampled datapoints of the full dataset (which is the description of 10-fold cross-validation)
Now, each of these iterations is done three times: one time for each parameter value.
Hence I end up with 30 iterations (10-fold cv basically, but for each parameter once) and pick the parameter which on average performed best. I am aware that this chosen model has not YET been tested on new data. However during all the iterations, since test and training set have been disjunct and independent, it's performance has been evaluated on this unseen test set. So why is this approach incorrect (as in comparison with nested cv with an inner and outer loop)?