There are many posts about how to do it but I miss one particular aspect, namely if the feature selection should be done for each validation set based on its accuracy (Case A) or as a mean accuracy of all sets (Case B).
I have 5 cross-validation sets. For each I do a feature selection (say backward selection) based on local accuracy. So, validation set 1 will identify variables Var1, Var2, Var3 to improve its accuracy. Validation set 2 will have Var1, Var4 and Var5 as selected variables. So, in fact, each validation set will identify its own 3 important variables. In the end I will have 15 variables in total. Should I take the most frequent ones then?
I identify my 3 variables based on the accuracy of all 5 cross-validation sets. So, if Var1 improves the mean accuracy of all 5 sets, it is included. In the end I will have 3 variables that improved the mean accuracy of my all 5 cross-validation sets.
Case B seems to me to be more accurate as we improving our performance over different sets. But in most posts I read that cross-validation should be in the outer loop, so the Case A should be the right one.
EDIT: The aim is to select features given that the classification algorithm, selection algorithm, hyper-parameters are static and will not change. So, I do not want to compare different classification algorithms, different selection algorithms.
EDIT2: I understand that I will need to train my final model with the selected features (that I selected in Case A or Case B) and the evaluate it on the test set. The question is, however, not about the final model. It is about how to select the features in the cross-validation because I do not understand which case (approach) is the right one.