I have read very well the awesome answers and suggestions by @cbeleites and @Dikran Marsupial here for nested CV but I am still confused about something:
Basically now I understand that nested CV is not used for model selection but rather for estimating the general performance of a certain model.
Having this said, what shall we do in order to tune hyper parameters if we cannot do that in nested CV ? because as also mentioned in the same link, we end up with K models (with K being the number of folds in the outer loop), and its not a good practice to choose the best out of those K models!
Shall we do regular cross validation again separately AFTER nested CV in order to tune hyper parameters ? (Although this is what the inner loop of the nested cv is about). I have read something similar in this thread
If we want to report the general performance of this model with nested CV: Can the mean/std scores of all the folds of the outer loop resemble 'testing' performance and the mean/std scores of all the folds of the inner loop resemble 'validation'? I used to do this when using 'regular' (not nested) cross validation:
- split the data into 80% training and 20% testing
- do cross validation on the 80% training which internally does training/validation splits with hyper parameter tuning, and report 'validation' performance
- choose the best model that contributed to the best score in the cross validation and use it to train again on the whole training data, then test on the 20% testing and report 'testing' performance.
However, this 80/20 split procedure is done several times (specifically K times) in the nested CV strategy.
So to make a long story short, can I achieve this effect with nested CV (my main goal is to report validation and testing performances)?
Many thanks in advance.