I spent quite a few hours trying to understand nested cross-validation and try and make an implementation myself — I'm really uncertain if I am doing this right, and I am not sure how to test if I am doing it right other than ask experts if I am indeed doing it right.
I am trying optimize the hyperparameters of my feature selector. Initially, I fell into the trap of introducing selection bias in my evaluation of my model, similar to what was described here, namely, I was trying to optimize my hyperparameters using the same data I selected features with.
Here is my understanding of nested cross-validation mixed with how I am trying to use it: (using the number of folds = 5 for simplicity):
- Split all the data into 5 sets, four of which will be used to train, one of which will be used to test.
- Create the parameter grid I want to search over, and do a grid search over the parameter grid with 5-fold cross validation, using the "outer" training data, and store the parameter set that best maximizes the ROC score (using sklearn's GridSearchCV, which is why this is all in one step).
- Now that I have the best parameters for my random forest (the best for this iteration), fit that Random Forest model to the "outer" training data, and transform the outer training, and test data (i.e., use the best feature selector obtained by (2) to actually feature select).
- Fit my "main" model (logistic regression), using the transformed outer data
- Using the transformed outer testing data, use the fitted logistic regression model made in (4) to make predictions, then find the ROC score of those predictions.
- Store the best parameters found in (3) and the score found in (5), and start again from (1), using a different set from the original 5 as the testing set, and the other 4 as training sets.
- At the end of it all, the parameters associated with the highest score in (6) should be used as the parameters for my Random Forest feature selection process.
During my research on this topic, I saw the phrases "internal validation is not enough!" and "don't validate with data used to train" often. By internal validation, I think they're referring to what I do in (2), and by doing a larger cross validation on the results I get from it, I think I am addressing the "not enough" part and also using different data to train and fit by having this outer validation. I think I am also addressing selection bias by repeating the feature selection each iteration of the outer cv.
Am I missing something? When looking at examples of other people doing this, it seems like they use nested cross-validation for either optimizing hyperparameters or to feature select. That makes me feel that I should have another nest somewhere, but I don't see where it would work. Also the "random" part of Random Forests is giving me uncertainty that I am doing anything generalizable rather than something that's entirely dependent on chance.
Also, when people say "use the inner CV to pick parameters", I am not really sure what that means, because the inner CV is run multiple times, isn't it?