I am trying to implement a sequential backwards selection algorithm to select features with cross validation. I find this straightforward when it comes to the steps:
- start with n features
- remove feature n, train model, replace feature n, do the same with feature n-1, etc.
- once iterated through all n features, remove feature with highest score, as it is the least impactful
- repeat steps 1-3 with n-1 features
- do this until all but one feature has been removed to acquire ranked list of features by importance
However when considering cross validation I am confused as to how this should be implemented. I have in mind two approaches:
cross validate first - split data into k folds and perform feature selection independently on each of these folds. I am not sure how I would select the features, perhaps a weighting scheme where the features are weighted based on their rank and averaged across all folds.
for removal of each feature, split into k folds and calculate mean of these k folds to get a score. After performing steps 1-3, remove the feature with the highest score.
It seems like the first option would be the correct way of doing it, but I cannot come up with why option 2 would be incorrect. I have looked into libraries that implement this but was unable to figure out how exactly they worked with cross validation.