My question is somewhat related to this question I run k-fold inner cross validation (k = 7), and outer-validate the best found model on a hold-out sample m times (m = 99). At each m I use unique training/validation/hold-out data. I end up with 99 entries, which include inner-cv average of the minimized metric, model hyperparameters tuned, chosen inputs on the one hand, and the minimized metric value for an outer-cv sample.
What can follow was advised to be (citation):
the difference between the estimates in the inner vs. outer CV. This gives an indication whether there might be problems with overfitting in the hyperparameter optimization.
How stable are the reported hyperparameters over the 10 folds of the outer CV?
How stable is the performance over the surrogate models of the outer cross validation. You can draw conclusions on this only if you either have enough test cases in each of the folds, or do iterated/repeated cross validation for the outer loop.
So, I have tools to understand if the ML method chosen, coupled with the inner CV, produces stable and reproducible results on hold-out test sets.
Assume now that I repeated this procedure for 90 different (approx. independent) data arrays. I deal with 90 summary dataframes (each covering the 99 outer-cv results). I am kind of trapped into the need to pick up a best 1 of 90 outcomes which is supposed to bring the lowest generalization error on some future data. I am afraid that picking a best outer-cv evaluation metric (i.e., median value over 99 sets) will result in the problem of model selection bias: choosing models that best perform on the hold out set (outer-cv). That can lead to poor true generalization error.
So the question comes here: if I don't pick the best evaluation metric on the outer-cv data, am I only left with the cited tools that can help me understand the homogeneity of params/results for inner vs. outer-cv data? And the one models that behave evenly on inner and outer data (although not necessarily producing the lowest hold out performance) would be the best estimator of a robust model?