How are inner loop and outer loops used to evaluate and build a machine learning model?

Question

Ok. First off, I want to say that I know that there have been many questions that have been asked regarding cross validation or nested cross validation ( Nested cross validation for model selection and the article by Calbot). Unfortunately, I still feel confused. Below is a toy example problem along with two solutions, one that involves traditional cross validation, and the other that involves nested cross validation.

Assume a data set, Y, that we have divided into training and test. Furthermore, we have divided our training into 5 folds.

I want to try to build the best model that generalizes well to unknown data. I have picked two algorithms, a Random Forest and a Support Vector Machine. In addition to the different algorithms, I also know that there may be 2 sets of different features I want to include or exclude AND 2 hyperparameters for each algorithm that I want to tune. In order to find the best model among the many different combinations of algorithms, hyperparameters, and features I could include while building a model, my plan would look like this:

1) Use a grid search and cross validation to assess which model building process is most likely to generalize well to unseen data (The total number of models built here is 5 (the number of folds) x 2 (the number of algorithms I want to try) x 2 (the sets of features I could include for any model) x 2 (the number of hyperparameters per algorithm I would like to tune) = 40 different models). Let's assume that among all these different combinations, the model with the lowest CV error is a random forest of 200 trees with feature set 1.

2) After assessing which combination of decisions is most likely to generalize well to unknown data, I pick said combination and train it on the entire training set.

3) In order to determine how well this new model will do, I then use the test data to determine the test error.

This process seems reasonable to me for one main reason: when picking a process to pick a model, I am assessing the models not on how they are performing on the data on which they were trained. Still, I have read and heard that nested cross validation is necessary. From what I understand that process would look something like this (?)

1) Tune your hyperparameters and feature selection for each algorithm on an inner loop (for the purposes of this example, if Outer Loop 1 = (Training: Folds 1,2,3,4; Test: Fold 5), then Inner Loop = (Folds 1,2,3, 4).

2) Use the tuned algorithms from this inner loop in the outer loop to...what?

3) Pick the process to build a model from the … loop and train on...?

4) Use the test set to assess performance

My questions:

1) Is nested cross validation really necessary, given that I have both a test set to determine how the model will generalize and a cross validation process to assess which combination of model building decisions is most likely to generalize well to the test set?

2) If nested cross validation is necessary, then I am consolidating certain decisions of the model building process to the inner loop? Which ones and why?

3) What information do I gain from the inner loop that I apply to the outer loop? If I am just tuning hyper parameters in the inner loop, then am I picking the algorithm in the outer loop?

A final note: I realize that there is a lot ambiguity in the machine learning community. For any answer submitted, please help me understand what the specific process would look like (don't just say I'm wrong and what I misunderstand).

Answer 1

It's not mandatory when you do the the other way correct as you've explained in the first part, because conceptually at every step models are competing with each other, so the type of the algorithm doesn't matter. Algorithms $A$, $B$ with hyper-parameter sets $\mathcal{H}_A$ and $\mathcal{H}_B$ can be selected by making the list of models as $$\{(A,\mathcal{h_a}_1),(A,\mathcal{h_a}_2),\dots,(B,\mathcal{h_b}_1),(B,\mathcal{h_b}_1),\dots\}$$ and inputting them into the usual CV loop. Note that, nested and usual CV might give different outputs, but size of the data decreases the variance of decision.

Just to complicate things, think of logistic regression and neural networks, where the former can be thought of as a special case of neural net (if used the same optimizer & loss). So, there can be overlaps between different algorithms, which indicates that the boundaries of CV might be softer.

Example outside ML: Imagine a national competition that we want to select the best student in all the country.

Nested CV: Select the best student from the city, then make them race for the state, and then make them race for the whole country.

Usual CV: Put all students over the country to the contest, and pick the best one.

But, in the end, you'll still pick the best one, assuming your evaluation metric is unbiased.