correct feature selection for stable feature subset I was just reading "Resampling Strategies for Model Assessment
and Selection" by Richard Simon (Springer), and he says the following on page 183:

"Data analysts are sometimes tempted to use the similarities and
  differences among the classifiers developed on the resampled training
  sets for purposes of model selection. This happens particularly often
  with regard to feature selection. That is, a defined feature selection
  is used for classifier development in each training set. The analyst
  is often surprised at how much the sets of selected features differ
  among the training sets and they are tempted to use for the final
  classifier only those features that were consistently selected in most
  training sets. That, however, represents a completely new algorithm
  for classifier development. When one is using resampling for
  estimating the true error rate of a classifier, then all aspects of
  the algorithm used for developing the classifier must be repeated in
  each resampled learning set. If part of the algorithm involves
  determining which variables to select based on their frequency of
  selection in resampling the data, then the resampling process must be
  performed for variable selection for each learning set of the
  resampling loop used for error estimations."

Here I have two questions:
1) If I understand correctly, the gist of his point is simply that if we select this "stable" subset of features over several folds, that obviously we don't have any model performance estimation at this point, and hence would have to re-run in our nested cross-validation the outer loop for a performance estimation again - is my understanding correct?
2) usually with regards to nested CV, it is only discussed that the inner loop is for model selection and the outer for model evaluation. However, what about feature selection? Is feature selection only performed on (each separate fold of) the inner loop only?
Many thanks
 A: 1) yes
2) yes, feature selection should only be performed in the inner loop.
I like to think of feature selection and model selection by cross validation as data-driven (i.e. by the inner CV results) optimization of the model. It really doesn't matter whether the hyperparameter you optimize are the input features or the modeling algorithm or the preprocessing steps or whatever. As soon as you use the inner CV results to select/optimize the model, nested validation is necessary.

Another comment on the quoted practice: Make sure selecting only "stable" features is sensible at all for your kind of data.
If you have colinearities in your data (redundant features), selecting only the stable features may result in much worse models. For example I work with vibrational spectra, a type of data that for physical reasons has several signals showing up from the same substance. There may be some models using signals A + B, whereas others use A + C with B and C being in theory equivalent to each other but not equivalent to A. One model selecting B and the other C for prediction may be a caused by slight random variations in the training sets that cause either B or C to be slightly preferred and therefore selected over the other. Throwing out B and C because they are not used consistently would not be a good idea at all in this situation.
