0
$\begingroup$

I am trying to implement a sequential backwards selection algorithm to select features with cross validation. I find this straightforward when it comes to the steps:

  1. start with n features
  2. remove feature n, train model, replace feature n, do the same with feature n-1, etc.
  3. once iterated through all n features, remove feature with highest score, as it is the least impactful
  4. repeat steps 1-3 with n-1 features
  5. do this until all but one feature has been removed to acquire ranked list of features by importance

However when considering cross validation I am confused as to how this should be implemented. I have in mind two approaches:

  1. cross validate first - split data into k folds and perform feature selection independently on each of these folds. I am not sure how I would select the features, perhaps a weighting scheme where the features are weighted based on their rank and averaged across all folds.

  2. for removal of each feature, split into k folds and calculate mean of these k folds to get a score. After performing steps 1-3, remove the feature with the highest score.

It seems like the first option would be the correct way of doing it, but I cannot come up with why option 2 would be incorrect. I have looked into libraries that implement this but was unable to figure out how exactly they worked with cross validation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.