Say I have a set of classifier models, each generated using feature selection inside a repeated k-fold cross-validation. Each classifier model is generated using a different set of regularization parameters or hyperparameters.
I understand that choosing the 'best' model of this set, i.e. one that yields best k-fold cross validation classication estimate could produce an optimistically biased estimate of generalized performance. However is bias avoided if the final performance estimate, is based on a separate repeated k-fold cross validation using the features and hyperparameters selected above?
I have found this procedure (10 folds, 10 repetitions) works well in practice (model appears stable on genuinely unseen data) on a data set with Cases > Features however I wonder if any remaining bias could be considered unacceptable? I suspect this procedure is less acceptable in the case where Features >> Cases
My question is related to Training with the full dataset after cross-validation?
Apologies if this question appears ignorant or repeats material discussed elsewhere.