I was just reading "Resampling Strategies for Model Assessment and Selection" by Richard Simon (Springer), and he says the following on page 183:
"Data analysts are sometimes tempted to use the similarities and differences among the classifiers developed on the resampled training sets for purposes of model selection. This happens particularly often with regard to feature selection. That is, a defined feature selection is used for classifier development in each training set. The analyst is often surprised at how much the sets of selected features differ among the training sets and they are tempted to use for the final classifier only those features that were consistently selected in most training sets. That, however, represents a completely new algorithm for classifier development. When one is using resampling for estimating the true error rate of a classifier, then all aspects of the algorithm used for developing the classifier must be repeated in each resampled learning set. If part of the algorithm involves determining which variables to select based on their frequency of selection in resampling the data, then the resampling process must be performed for variable selection for each learning set of the resampling loop used for error estimations."
Here I have two questions:
1) If I understand correctly, the gist of his point is simply that if we select this "stable" subset of features over several folds, that obviously we don't have any model performance estimation at this point, and hence would have to re-run in our nested cross-validation the outer loop for a performance estimation again - is my understanding correct?
2) usually with regards to nested CV, it is only discussed that the inner loop is for model selection and the outer for model evaluation. However, what about feature selection? Is feature selection only performed on (each separate fold of) the inner loop only?