How do you properly use feature importance values from tree ensemble methods for feature selection without biasing your validation metric?
I have a colleague at work that basically calculates permutation importance on our validation set and drops those features that appear at the bottom of the list. But, she also uses that same validation set to determine if it improved generalization or not. I argued to her that as soon as you do that your validation score becomes biased because you've used the knowledge of what particular features perform best on that data to select a model and you're also using that same set simultaneously to evaluate that model.
On the other hand, I'm failing to see a better alternative. I recognize that using training permutation or feature importance can have the opposite problem where you leave in features that the model has happened to "learn from" but are not actually conducive to generalization.
What is the correct way to approach this?