1
$\begingroup$

I have read several threads and articles about the need to use nested cross validation when, for example, choosing between different machine learning algorithms. Here are two describing the use of nested cross validation in algorithm selection: https://sebastianraschka.com/faq/docs/evaluate-a-model.html, https://towardsdatascience.com/a-short-introduction-to-model-selection-bb1bb9c73376.

I am aware that the process of nested cross validation in selecting between, for example different algorithms, is as follows (for the case of k fold CV):

  1. The inner loop CV uses the 'outer loop' training sets, to find the optimal hyperparameter. The optimal hyperparameter is chosen based on its test performance on the 'inner loop' test folds. Here we loop over a selection of different hyperparamter values. If we performed 5 fold CV in the inner loop, then each hyperparameter's 'score' is the average of the scores from testing it on the 5 'inner loop test' folds.
  2. The 'inner loop' gives us our best hyperparameter. We then train a model on the 'outer loop' training data (which we earlier split inside the 'inner loop') and calculate its test performance on the 'outer loop' test fold.
  3. This entire process is repeated for as many 'outer loop' folds we are using. So, if we were doing 5 fold 'outer loop' CV, we could (depending on model stability) end up with 5 different hyper parameters.
  4. Once this is repeated for these 'outer loop' folds, we take the average of all the 'outer loop' test fold scores to get the overall test performance of that particular algorithm.
  5. This is repeated for all the algorithms we are comparing. The one with the best score is taken forward.

I know the purpose of separating the model selection (done in the inner loop) and performance estimation (done on the outer loop) is to avoid biasing the results somehow, but i don't understand why. I have looked at several threads on here but cant get my head around it (e.g. Feature selection and cross-validation , Training with the full dataset after cross-validation?)

Take the following example of where normal (say 5 fold) cross-validation is used. This would involve the following steps:

  1. Split data into 5 folds
  2. Select the first algorithm to test.
  3. Take a given value form the selection of hyperparameter values we wish to test, and train it on the 4 training folds.
  4. Calculate a test score using the remaining test fold.
  5. Repeat for all 5 folds.
  6. Average all 5 test scores to get the average test score for that hyperparameter.
  7. Repeat this for all values of the hyperparameter. The one with the best test score is carried forward.
  8. Repeat this for all the algorithms we wish to test.
  9. Compare the average test scores for each algorithm and the one with the best test score gets carried forward.

I can't seem to understand why the above method, which doesn't require nested cross validation is not sufficient?

I guess the question is, why do we need to separate the model selection and performance estimation steps into the inner and outer loops? I can't see the benefits of this compared to doing both at the same time, which is what the second set of steps described does.

UPDATE I've come across a video which explains the reasoning very well: https://www.youtube.com/watch?v=DuDtXtKNpZs

$\endgroup$
2
  • $\begingroup$ @Firebug Thanks but not really. It more talks about why the inner loop and outer loops do, which i understand. It also discusses selecting the model at the end. I understand that the purpose of nested-cross validation is not to select an exact model, but to essentially generate each algorithm or models test performance in order to compare against the others, and select the best one. I was hoping for an explanation of why the outer loop is required, rather than what is does. More so, why do we need to separate the model selection and performance estimation into two separate loops? $\endgroup$
    – Sean
    Commented May 7, 2020 at 18:16
  • $\begingroup$ @Firebug I've updated the end of my question to hopefully make what I'm asking a bit clearer. $\endgroup$
    – Sean
    Commented May 7, 2020 at 18:20

0