1
$\begingroup$

I am planning to implement nested cross-validation, but just had a question about its operation. I know there are lots of posts about nested cv, but none of them (as far as I understand) address my mis-understanding about the process.

Context: I found the illustration (shown below) in the following blog to be the simplest explanation of what is going on: here.

Image of Nested CV

Question: How does the outer loop work if each of the inner loop cv processes yield a different optimal set of hyperparameters?

To explain what I mean, I will refer to the image above which has 3 folds in the outer loop, which I will refer to as Fold 1, 2, and 3 respectively.

For the first iteration of the outer loop, we use Fold 1 as the holdout test set and we pass in Folds 2 & 3 for (Kfold cv) hyper parameter tuning in the inner loop. Let us say this yields a certain set of optimal hyper-parameters: hyperparameter set A. Then we train a model with all of Folds 2 & 3 as training data, using set A of hyperparams, and test on Fold 1 - we get accuracy A.

Now for the next iteration of the outer loop, use Fold 2 as the test holdout set and pass in Folds 1 & 3 to the inner loop cv process. Let us say this yields a different set of optimal hyper-parameters: hyperparameter set B. Then we train a model with all of Folds 1 & 3 as training data, using set B of hyperparams, and test on Fold 2 - we get accuracy B.

For completeness, we can repeat the above for third iteration of outer loop and obtain some new set of optimal hyper-parameters: hyperparameter set C. Then we train a model with all of Folds 1 & 2 as training data, using set C of hyperparams, and test on Fold 3 - we get accuracy C.

This is what I am confused about:

  • We now have three different models/sets of hyperparameters. How has the outer loop helped us to evaluate the performance in a general setting?
  • Can I simply take the average of accuracy A, B, and C? If so, what does that represent?

I hope this question makes sense. I can try to elaborate if required.

Improve this question
$\endgroup$
2
  • $\begingroup$ Your question makes perfect sense. I'm looking for an answer for exactly the same question for about a couple of hours without success. As you have explained clearly, the inner loop might select different hyperparameters at each iteration. Hope somebody will answer. Have you found an answer in the mean time? $\endgroup$
    – Sanyo Mn
    Apr 11, 2022 at 16:00
  • $\begingroup$ Hi, thanks for post (nice to hear that other people are thinking about this same problem). Unfortunately, I haven't found any conclusive answer. In terms of aggregating the results (accuracies A, B, and C as described in the post), I was simply advised to either take the mean or median of those results depending on what I thought the distribution of the results to be. However, I am still unsure of what to do in terms of reporting results if/when all hyper parameter sets are different. Maybe someone will eventually post an answer. $\endgroup$
    – Rocky the Owl
    Apr 11, 2022 at 20:21

2 Answers 2

Reset to default
2
$\begingroup$

In nested CV, the average of the errors of the outer loop can be seen as the expected error of the model building process (including the hyperparameter tuning).

Let me try to clarify this. First, let us consider a non-nested CV case where the number of folds is 5. Suppose that in each fold a tree-based model is built with the default hyperparameters. In this situation, even though we are using the same learning algorithm with the same parameters, in each fold a different model will be built since the training data used by the algorithm is different in each fold. So, we have 5 different models and a different error value for each model. Now, you can ask a similar question, what does the average error (over 5 folds) represent? In this case the average error represents the expected error of this particular model building process.

Similarly, in the nested CV case, the average error represents the expected error of the model building process. Different from the non-nested CV case, in nested CV the model building process includes a grid search over the hyperparameters. Hope that makes sense.

Improve this answer
$\endgroup$
1
  • $\begingroup$ "in nested CV the model building process includes a grid search over the hyperparameters" isn't non-nested CV also including a grid search over hyperparameters? $\endgroup$
    – Sextus Empiricus
    May 5 at 15:10
0
$\begingroup$

@SanyoMn already answered the question about pooling the results of the k outer folds.

 We now have three different models/sets of hyperparameters. How has the outer loop helped us to evaluate the performance in a general setting?

The outer loop told us that the hyperparameter optimization heuristic in the inner loop did not yield stable results for the given data set.

Typically, we'd say that instability is a bad sign. However, depening on the situation at hand, the instability can also be harmless. That would be up to you to judge.

E.g. I've had a hyperparameter optimization where I observed instability that purely shifted part of the overall model complexity between preprocessing (EMSC based on principal components of a blank measurement subset of the training data) and the "actual" model (PLSR) but the overall complexity stayed constant. That is something I'm pretty relaxed about.
Whereas I'd been far more concerned if there had been a large uncertainty in the "optimal" model complexity between the outer folds (say, sometimes 5, sometimes 15 components).

Also potentially relevant:

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.