I am currently trying to understand how exactly to use nested k-fold cross validation for hyperparameter tuning / model selection. There is one aspect I really cannot get my head around.
I found this thread: Nested cross validation for model selection where the question was already asked but I feel it wasn't quite answered, or I might just not get it. Let's assume I have a nested cross validation where the outer loop is divided into K-folds so that I end up with K inner loops that are each divided into N-Folds. And also K-test folds.
From what I read online, nested CV works as follows:
- There is the inner CV loop, where we may conduct a grid search (e.g. running N-fold for every available model, e.g. combination of hyperparameters/features)
- There is the outer CV loop, where we measure the performance of the model that won in the inner fold, on a separate external fold.
At the end of this process we end up with K models (K being the number of folds in the outer loop). These models are the ones that won in the grid search within the inner CV, and they are likely different (e.g. SVMs with different kernels, trained with possibly different features, depending on the grid search).
Let's say, I've done a grid search for finding the best value of the regularization parameter C for my SVM. This I've done by calculating the average performance on the N-validation folds inside each inner loop. This needs to be done for all K inner loops. And so I end up with K slightly different values for C, one for each of my K inner loops. How exactly do I choose the best value for C from the K determined best values of the inner loops? I think there is a basic concept I don't understand.