correct feature selection for stable feature subset

Question

I was just reading "Resampling Strategies for Model Assessment and Selection" by Richard Simon (Springer), and he says the following on page 183:

"Data analysts are sometimes tempted to use the similarities and differences among the classifiers developed on the resampled training sets for purposes of model selection. This happens particularly often with regard to feature selection. That is, a defined feature selection is used for classifier development in each training set. The analyst is often surprised at how much the sets of selected features differ among the training sets and they are tempted to use for the final classifier only those features that were consistently selected in most training sets. That, however, represents a completely new algorithm for classifier development. When one is using resampling for estimating the true error rate of a classifier, then all aspects of the algorithm used for developing the classifier must be repeated in each resampled learning set. If part of the algorithm involves determining which variables to select based on their frequency of selection in resampling the data, then the resampling process must be performed for variable selection for each learning set of the resampling loop used for error estimations."

Here I have two questions:

1) If I understand correctly, the gist of his point is simply that if we select this "stable" subset of features over several folds, that obviously we don't have any model performance estimation at this point, and hence would have to re-run in our nested cross-validation the outer loop for a performance estimation again - is my understanding correct?

2) usually with regards to nested CV, it is only discussed that the inner loop is for model selection and the outer for model evaluation. However, what about feature selection? Is feature selection only performed on (each separate fold of) the inner loop only?

Many thanks

Answer 1

1) yes

2) yes, feature selection should only be performed in the inner loop.
I like to think of feature selection and model selection by cross validation as data-driven (i.e. by the inner CV results) optimization of the model. It really doesn't matter whether the hyperparameter you optimize are the input features or the modeling algorithm or the preprocessing steps or whatever. As soon as you use the inner CV results to select/optimize the model, nested validation is necessary.

---

Another comment on the quoted practice: Make sure selecting only "stable" features is sensible at all for your kind of data.
If you have colinearities in your data (redundant features), selecting only the stable features may result in much worse models. For example I work with vibrational spectra, a type of data that for physical reasons has several signals showing up from the same substance. There may be some models using signals A + B, whereas others use A + C with B and C being in theory equivalent to each other but not equivalent to A. One model selecting B and the other C for prediction may be a caused by slight random variations in the training sets that cause either B or C to be slightly preferred and therefore selected over the other. Throwing out B and C because they are not used consistently would not be a good idea at all in this situation.