Ok. First off, I want to say that I know that there have been many questions that have been asked regarding cross validation or nested cross validation ( Nested cross validation for model selection and the article by Calbot). Unfortunately, I still feel confused. Below is a toy example problem along with two solutions, one that involves traditional cross validation, and the other that involves nested cross validation.
Assume a data set, Y, that we have divided into training and test. Furthermore, we have divided our training into 5 folds.
I want to try to build the best model that generalizes well to unknown data. I have picked two algorithms, a Random Forest and a Support Vector Machine. In addition to the different algorithms, I also know that there may be 2 sets of different features I want to include or exclude AND 2 hyperparameters for each algorithm that I want to tune. In order to find the best model among the many different combinations of algorithms, hyperparameters, and features I could include while building a model, my plan would look like this:
1) Use a grid search and cross validation to assess which model building process is most likely to generalize well to unseen data (The total number of models built here is 5 (the number of folds) x 2 (the number of algorithms I want to try) x 2 (the sets of features I could include for any model) x 2 (the number of hyperparameters per algorithm I would like to tune) = 40 different models). Let's assume that among all these different combinations, the model with the lowest CV error is a random forest of 200 trees with feature set 1.
2) After assessing which combination of decisions is most likely to generalize well to unknown data, I pick said combination and train it on the entire training set.
3) In order to determine how well this new model will do, I then use the test data to determine the test error.
This process seems reasonable to me for one main reason: when picking a process to pick a model, I am assessing the models not on how they are performing on the data on which they were trained. Still, I have read and heard that nested cross validation is necessary. From what I understand that process would look something like this (?)
1) Tune your hyperparameters and feature selection for each algorithm on an inner loop (for the purposes of this example, if Outer Loop 1 = (Training: Folds 1,2,3,4; Test: Fold 5), then Inner Loop = (Folds 1,2,3, 4).
2) Use the tuned algorithms from this inner loop in the outer loop to...what?
3) Pick the process to build a model from the … loop and train on...?
4) Use the test set to assess performance
1) Is nested cross validation really necessary, given that I have both a test set to determine how the model will generalize and a cross validation process to assess which combination of model building decisions is most likely to generalize well to the test set?
2) If nested cross validation is necessary, then I am consolidating certain decisions of the model building process to the inner loop? Which ones and why?
3) What information do I gain from the inner loop that I apply to the outer loop? If I am just tuning hyper parameters in the inner loop, then am I picking the algorithm in the outer loop?
A final note: I realize that there is a lot ambiguity in the machine learning community. For any answer submitted, please help me understand what the specific process would look like (don't just say I'm wrong and what I misunderstand).