Nested cross validation: how does the outer loop work? I am planning to implement nested cross-validation, but just had a question about its operation. I know there are lots of posts about nested cv, but none of them (as far as I understand) address my mis-understanding about the process.
Context: I found the illustration (shown below) in the following blog to be the simplest explanation of what is going on: here.

Question: How does the outer loop work if each of the inner loop cv processes yield a different optimal set of hyperparameters?
To explain what I mean, I will refer to the image above which has 3 folds in the outer loop, which I will refer to as Fold 1, 2, and 3 respectively.
For the first iteration of the outer loop, we use Fold 1 as the holdout test set  and we pass in Folds 2 & 3 for (Kfold cv) hyper parameter tuning in the inner loop. Let us say this yields a certain set of optimal hyper-parameters: hyperparameter set A. Then we train a model with all of Folds 2 & 3 as training data, using set A of hyperparams, and test on Fold 1 - we get accuracy A.
Now for the next iteration of the outer loop, use Fold 2 as the test holdout set and pass in Folds 1 & 3 to the inner loop cv process. Let us say this yields a different set of optimal hyper-parameters: hyperparameter set B. Then we train a model with all of Folds 1 & 3 as training data, using set B of hyperparams, and test on Fold 2 - we get accuracy B.
For completeness, we can repeat the above for third iteration of outer loop and obtain some new set of optimal hyper-parameters: hyperparameter set C. Then we train a model with all of Folds 1 & 2 as training data, using set C of hyperparams, and test on Fold 3 - we get accuracy C.
This is what I am confused about:

*

*We now have three different models/sets of hyperparameters. How has the outer loop helped us to evaluate the performance in a general setting?

*Can I simply take the average of accuracy A, B, and C? If so, what does that represent?

I hope this question makes sense. I can try to elaborate if required.
 A: In nested CV, the average of the errors of the outer loop can be seen as the expected error of the model building process (including the hyperparameter tuning).
Let me try to clarify this. First, let us consider a non-nested CV case where the number of folds is 5. Suppose that in each fold a tree-based model is built with the default hyperparameters. In this situation, even though we are using the same learning algorithm with the same parameters, in each fold a different model will be built since the training data used by the algorithm is different in each fold. So, we have 5 different models and a different error value for each model. Now, you can ask a similar question, what does the average error (over 5 folds) represent? In this case the average error represents the expected error of this particular model building process.
Similarly, in the nested CV case, the average error represents the expected error of the model building process. Different from the non-nested CV case, in nested CV the model building process includes a grid search over the hyperparameters. Hope that makes sense.
A: @SanyoMn already answered the question about pooling the results of the k outer folds.

 We now have three different models/sets of hyperparameters. How has the outer loop helped us to evaluate the performance in a general setting?

The outer loop told us that the hyperparameter optimization heuristic in the inner loop did not yield stable results for the given data set.
Typically, we'd say that instability is a bad sign. However, depening on the situation at hand, the instability can also be harmless. That would be up to you to judge.
E.g. I've had a hyperparameter optimization where I observed instability that purely shifted part of the overall model complexity between preprocessing (EMSC based on principal components of a blank measurement subset of the training data) and the "actual" model (PLSR) but the overall complexity stayed constant. That is something I'm pretty relaxed about.
Whereas I'd been far more concerned if there had been a large uncertainty in the "optimal" model complexity between the outer folds (say, sometimes 5, sometimes 15 components).
Also potentially relevant:

*

*https://stats.stackexchange.com/a/65156/4598

*https://stats.stackexchange.com/a/245169/4598
