1
$\begingroup$

There are many posts about how to do it but I miss one particular aspect, namely if the feature selection should be done for each validation set based on its accuracy (Case A) or as a mean accuracy of all sets (Case B).

Example.

Case A:
I have 5 cross-validation sets. For each I do a feature selection (say backward selection) based on local accuracy. So, validation set 1 will identify variables Var1, Var2, Var3 to improve its accuracy. Validation set 2 will have Var1, Var4 and Var5 as selected variables. So, in fact, each validation set will identify its own 3 important variables. In the end I will have 15 variables in total. Should I take the most frequent ones then?

Case B:
I identify my 3 variables based on the accuracy of all 5 cross-validation sets. So, if Var1 improves the mean accuracy of all 5 sets, it is included. In the end I will have 3 variables that improved the mean accuracy of my all 5 cross-validation sets.

Case B seems to me to be more accurate as we improving our performance over different sets. But in most posts I read that cross-validation should be in the outer loop, so the Case A should be the right one.

EDIT: The aim is to select features given that the classification algorithm, selection algorithm, hyper-parameters are static and will not change. So, I do not want to compare different classification algorithms, different selection algorithms.

EDIT2: I understand that I will need to train my final model with the selected features (that I selected in Case A or Case B) and the evaluate it on the test set. The question is, however, not about the final model. It is about how to select the features in the cross-validation because I do not understand which case (approach) is the right one.

Improve this question
$\endgroup$
8
  • $\begingroup$ You want to control your selection with a complexity parameter that is selected with cross validation. For example, in LASSO selection, the complexity parameter is lambda, and this is chosen using cross validation. The variables or fit parameters themselves are not chosen by cross validation. $\endgroup$
    – Matthew Drury
    Commented Jul 9, 2017 at 21:55
  • $\begingroup$ @MatthewDrury if there is LASSO, why are there forward and backward feature selection techniques? Where can we apply them? $\endgroup$
    – Alina
    Commented Jul 9, 2017 at 22:16
  • $\begingroup$ Forward/backward selection have hyper-parameters related to the complexity of the solution (e.g. using AIC/BIC as a stopping criterion, or setting a maximum number of variables to select). The cross-validation procedure is used to estimate the predictive performance of the final model. The actual final model, which contains a set of selected features, is created by applying the feature selection + modelling algorithms on the full dataset. If in addition to performance estimation you also want to select the best hyper-parameter configuration, a method like nested cross-validation should be used $\endgroup$
    – George
    Commented Jul 9, 2017 at 22:23
  • $\begingroup$ @George are you suggesting to select features on the full dataset and then apply cross-validation to estimate the predictive performance? As I described, I just want to select features and as far as I understood one would do it with a cross-validation looking if by adding one variable, the accuracy would improve or not. I mean, you can always do it mainly just by removing the feature and see if your mean accuracy improved (ritchieng.com/machine-learning-cross-validation). So, in this example there is no need for AIC/BIC $\endgroup$
    – Alina
    Commented Jul 9, 2017 at 22:47
  • $\begingroup$ Yes, unless you also want to optimize over hyper-parameters values of your algorithms. If for instance you want to try out various feature selection algorithms and/or hyper-parameter values (e.g. different values of \lambda for LASSO), as well as different classification algorithms, you should have a look into nested cross-validation in order to perform simultaneous performance estimation and hyper-parameter optimization. $\endgroup$
    – George
    Commented Jul 9, 2017 at 22:48

1 Answer 1

Reset to default
1
$\begingroup$

(B) "Var1 improves the mean accuracy of all 5 sets" is usually the way to go. The reason why B is better than A is that your selection depends crucially on being able to find that performance improves. That needs very precise estimates of performance, which in turn depends on the number of cases tested.

The validation sets in A are only 1/5 of the total number of tested cases in B and therefore the variance due to the finite number of test cases would be more than 2x as much than for the test results in B.

If you'd like to have a check on the stability (i.e. are the same features selected for different splits), you can do iterated/repeated cross validation with B.

Having done the feature selection, IMHO you do need a proper estimate of generalization error (i.e. an outer loop of cross validation or another, still independent test set). I cannot think of a single application where this wouldn't be needed, even if you won't be able to improve with the given data. In case you are interested in the estimates themselves rather than in the predictive performance, you can directly measure this, but again this should be done independent of the optimization (e.g. repeated/iterated cross validation for the outer loop).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.