Model Selection Problem Using Nested Cross-Validation in Presence of Several Alternatives

Question

My question is somewhat related to this question I run k-fold inner cross validation (k = 7), and outer-validate the best found model on a hold-out sample m times (m = 99). At each m I use unique training/validation/hold-out data. I end up with 99 entries, which include inner-cv average of the minimized metric, model hyperparameters tuned, chosen inputs on the one hand, and the minimized metric value for an outer-cv sample.

What can follow was advised to be (citation):

the difference between the estimates in the inner vs. outer CV. This gives an indication whether there might be problems with overfitting in the hyperparameter optimization.

How stable are the reported hyperparameters over the 10 folds of the outer CV?

How stable is the performance over the surrogate models of the outer cross validation. You can draw conclusions on this only if you either have enough test cases in each of the folds, or do iterated/repeated cross validation for the outer loop.

So, I have tools to understand if the ML method chosen, coupled with the inner CV, produces stable and reproducible results on hold-out test sets.

Assume now that I repeated this procedure for 90 different (approx. independent) data arrays. I deal with 90 summary dataframes (each covering the 99 outer-cv results). I am kind of trapped into the need to pick up a best 1 of 90 outcomes which is supposed to bring the lowest generalization error on some future data. I am afraid that picking a best outer-cv evaluation metric (i.e., median value over 99 sets) will result in the problem of model selection bias: choosing models that best perform on the hold out set (outer-cv). That can lead to poor true generalization error.

So the question comes here: if I don't pick the best evaluation metric on the outer-cv data, am I only left with the cited tools that can help me understand the homogeneity of params/results for inner vs. outer-cv data? And the one models that behave evenly on inner and outer data (although not necessarily producing the lowest hold out performance) would be the best estimator of a robust model?

Answer 1

I am kind of trapped into the need to pick up a best 1 of 90 outcomes which is supposed to bring the lowest generalization error on some future data.

Do not use the outermost validation for selection. It is exclusively for estimating generalization error...

I am afraid that picking a best outer-cv evaluation metric (i.e., median value over 99 sets) will result in the problem of model selection bias: choosing models that best perform on the hold out set (outer-cv). That can lead to poor true generalization error.

... for exactly this reason (overoptimistic bias)

am I only left with the cited tools that can help me understand the homogeneity of params/results for inner vs. outer-cv data?

There are other possibilities. More basic ones than measures concerned with homogeity would be median/average performance and spread of observed performance.

The outer validation in your case is just a repeated measurement of some value (here: some figure of merit of the model performance) and you can (and IMHO should) treat it as such.
This is like measuring the frequency of a pendulum in the basic experimental physics labwork practicum or repeated titration in an analytical chemistry labwork practicum. You do not select the middlemost or smallest or largest of the resulting values but use the repetitions to get a better estimate of the underlying true value by using e.g. the mean and calculating a confidence interval for the mean.
Now with your outer validation, the repetitions may only be more-or-less independent (and you may know of confounders that complicate the situation), so in practice you may not know the equivalent number of truly independent repetitions (effective sample size for the outer validation) so confidence interval calculation may not be possible. However, you can still summarize the observations e.g. by giving the number of repetitions, and mean/median as well as some indication of the variability (standard deviation, 5th to 95th percentile, ...)

And the one models that behave evenly on inner and outer data (although not necessarily producing the lowest hold out performance) would be the best estimator of a robust model?

If the spread is low, i.e. results are stable, there is no need to choose: take any of the models, and you'll get this performance:

• If the optimization went well, you'll have good and stable models, so there is no dilemma here.
• Keep in mind that some variance is due to the finite number of test cases. This is IMHO a good reference for deciding whether variance is low or high.

• If the model parameters are unstable because there are several possibilities of getting equivalent performance, the performance estimations will be stable, and again there is no dilemma.

• If the observed performance in the outer validation repetitions do vary widely (i.e. more than the expected variance for stable predicitons tested with varying data sets of size $n_{test}$), you know that the optimization is not (or not always) successful.
  In that case, it doesn't make sense to accept any of the models. Rather you should go back and improve your modeling. Yes, you can be lucky and get a good model by unstable training. But typically you are not, and on average, unstable models perform badly (that's why ensemble models help in this situation).

• If you decide to keep the one apparently best model for further use, you can do that if you use yet another independent data set for estimating its generalization error. (You just introduced another level of model selection.) Again, the same principles apply: if that independently measured performance is noticably worse (i.e. "in the field" of the other 90 models) you probably didn't get a better model, but just skimmed variance.