I am trying to implement a sequential backwards selection algorithm to select features with cross validation. I find this straightforward when it comes to the steps:

  1. start with n features
  2. remove feature n, train model, replace feature n, do the same with feature n-1, etc.
  3. once iterated through all n features, remove feature with highest score, as it is the least impactful
  4. repeat steps 1-3 with n-1 features
  5. do this until all but one feature has been removed to acquire ranked list of features by importance

However when considering cross validation I am confused as to how this should be implemented. I have in mind two approaches:

  1. cross validate first - split data into k folds and perform feature selection independently on each of these folds. I am not sure how I would select the features, perhaps a weighting scheme where the features are weighted based on their rank and averaged across all folds.

  2. for removal of each feature, split into k folds and calculate mean of these k folds to get a score. After performing steps 1-3, remove the feature with the highest score.

It seems like the first option would be the correct way of doing it, but I cannot come up with why option 2 would be incorrect. I have looked into libraries that implement this but was unable to figure out how exactly they worked with cross validation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.