2
$\begingroup$

Ok. First off, I want to say that I know that there have been many questions that have been asked regarding cross validation or nested cross validation ( Nested cross validation for model selection and the article by Calbot). Unfortunately, I still feel confused. Below is a toy example problem along with two solutions, one that involves traditional cross validation, and the other that involves nested cross validation.

Assume a data set, Y, that we have divided into training and test. Furthermore, we have divided our training into 5 folds.

enter image description here

I want to try to build the best model that generalizes well to unknown data. I have picked two algorithms, a Random Forest and a Support Vector Machine. In addition to the different algorithms, I also know that there may be 2 sets of different features I want to include or exclude AND 2 hyperparameters for each algorithm that I want to tune. In order to find the best model among the many different combinations of algorithms, hyperparameters, and features I could include while building a model, my plan would look like this:

1) Use a grid search and cross validation to assess which model building process is most likely to generalize well to unseen data (The total number of models built here is 5 (the number of folds) x 2 (the number of algorithms I want to try) x 2 (the sets of features I could include for any model) x 2 (the number of hyperparameters per algorithm I would like to tune) = 40 different models). Let's assume that among all these different combinations, the model with the lowest CV error is a random forest of 200 trees with feature set 1.

2) After assessing which combination of decisions is most likely to generalize well to unknown data, I pick said combination and train it on the entire training set.

3) In order to determine how well this new model will do, I then use the test data to determine the test error.

This process seems reasonable to me for one main reason: when picking a process to pick a model, I am assessing the models not on how they are performing on the data on which they were trained. Still, I have read and heard that nested cross validation is necessary. From what I understand that process would look something like this (?)

1) Tune your hyperparameters and feature selection for each algorithm on an inner loop (for the purposes of this example, if Outer Loop 1 = (Training: Folds 1,2,3,4; Test: Fold 5), then Inner Loop = (Folds 1,2,3, 4).

2) Use the tuned algorithms from this inner loop in the outer loop to...what?

3) Pick the process to build a model from the … loop and train on...?

4) Use the test set to assess performance

My questions:

1) Is nested cross validation really necessary, given that I have both a test set to determine how the model will generalize and a cross validation process to assess which combination of model building decisions is most likely to generalize well to the test set?

2) If nested cross validation is necessary, then I am consolidating certain decisions of the model building process to the inner loop? Which ones and why?

3) What information do I gain from the inner loop that I apply to the outer loop? If I am just tuning hyper parameters in the inner loop, then am I picking the algorithm in the outer loop?

A final note: I realize that there is a lot ambiguity in the machine learning community. For any answer submitted, please help me understand what the specific process would look like (don't just say I'm wrong and what I misunderstand).

$\endgroup$

1 Answer 1

0
$\begingroup$

It's not mandatory when you do the the other way correct as you've explained in the first part, because conceptually at every step models are competing with each other, so the type of the algorithm doesn't matter. Algorithms $A$, $B$ with hyper-parameter sets $\mathcal{H}_A$ and $\mathcal{H}_B$ can be selected by making the list of models as $$\{(A,\mathcal{h_a}_1),(A,\mathcal{h_a}_2),\dots,(B,\mathcal{h_b}_1),(B,\mathcal{h_b}_1),\dots\}$$ and inputting them into the usual CV loop. Note that, nested and usual CV might give different outputs, but size of the data decreases the variance of decision.

Just to complicate things, think of logistic regression and neural networks, where the former can be thought of as a special case of neural net (if used the same optimizer & loss). So, there can be overlaps between different algorithms, which indicates that the boundaries of CV might be softer.

Example outside ML: Imagine a national competition that we want to select the best student in all the country.

Nested CV: Select the best student from the city, then make them race for the state, and then make them race for the whole country.

Usual CV: Put all students over the country to the contest, and pick the best one.

But, in the end, you'll still pick the best one, assuming your evaluation metric is unbiased.

$\endgroup$
6
  • $\begingroup$ Thank you. I like your inner loop example, and it makes me wonder: 1) What part of the model building process am I evaluating during the inner loop? and 2) In this example, Isn't there a possibility that there may be a kid who didn't do well at an early stage that may have down well in national race? $\endgroup$
    – atirvine88
    Mar 28, 2020 at 12:17
  • $\begingroup$ 2. it's possible because we decide based on data, and the data we test them slightly changes between the two procedures. But, note that this doesn't mean that the non-nested version is superior. Not sure if I understood your first question though. $\endgroup$
    – gunes
    Mar 28, 2020 at 12:30
  • $\begingroup$ Let's focus on one inner loop: Folds 1-4. I don't understand what part of my model-building process I would be looking at when cross-validating at this stage. And given that I would repeat this process for the other inner loops (1345, 2345, etc.), I assume different assumptions/parts of my model building process may do better, e.g. the depth of the forest may be different on the inner loops. How do I pick? $\endgroup$
    – atirvine88
    Mar 28, 2020 at 12:55
  • $\begingroup$ @atirvine88 typically for each inner loop, you'll also divide your training set into folds. For example, Fold 1-4 is your new training + validation data. And, you don't use Fold 5, because it's your test set. Then, you apply another CV to the data containing Folds1-4, e.g. if K-fold with K = 3: InnerFold1, InnerFold2, InnerFold3. $\endgroup$
    – gunes
    Mar 30, 2020 at 20:10
  • 1
    $\begingroup$ two clarifying questions: let's say I am cross-validating a hyperparameter on the inner loop, i.e. folds 1-4. That means for each value of the hyperparameter I am creating four models. I then use the hyperparameter that does best, create a model using folds 1-4 and test on fold 5. If I do this for each inner fold, what happens if the hyperparameter is different across the inner folds? Also, assuming that the tuned model is the same for each inner loop, is the outerloop then be used to compare optimally tuned algorithms (I have a holdout test for final test model performance)? $\endgroup$
    – atirvine88
    Mar 30, 2020 at 21:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.