Lets assume for example that I am attempting to predict a binary outcome using p predictors in which n>p with methods including a LASSO Regression, a Logistic Regression and SVM with an RBF kernel. Lets also assume that I plan to use nested cross-validation for model selection and generalization error estimation.
My question is:
Should we estimate performance in the outer loop only for the best model? What if I estimate performance in the outer loop for the logistic regression model, the best LASSO model and best SVM model from the inner loop? Will then selecting one of these 3 'best' models for production based on the generalization error of the outer loop lead to bias? It is sometimes the case that if a more interpretable model such as a logistic model has only a slight decrease in performance compared to a more complex model, one might prefer the more interpretable model at the cost of some performance - hence, the motivation to estimate generalization error for more than just the best performing model from the inner loop.