1
$\begingroup$

How do you properly use feature importance values from tree ensemble methods for feature selection without biasing your validation metric?

I have a colleague at work that basically calculates permutation importance on our validation set and drops those features that appear at the bottom of the list. But, she also uses that same validation set to determine if it improved generalization or not. I argued to her that as soon as you do that your validation score becomes biased because you've used the knowledge of what particular features perform best on that data to select a model and you're also using that same set simultaneously to evaluate that model.

On the other hand, I'm failing to see a better alternative. I recognize that using training permutation or feature importance can have the opposite problem where you leave in features that the model has happened to "learn from" but are not actually conducive to generalization.

What is the correct way to approach this?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

You're correct to think that using the validation set for variable selection is in essence training on that data. Your colleague should only use the validation set as a means to monitor for over fitting or to compare models for selection.

If you were going to do variable selection properly I would imagine the procedure would look as follows:

  • Fit a tree ensemble and examine feature importance
  • Define some criteria in which you keep features based on the importance
  • Refit the model based on those features
  • Predict on the hold outset with the new model

This puts the variable selection procedure within the model fitting procedure.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for the response. To be clear, when you say keep features based on importance, these importance values would be found from the training data right? $\endgroup$
    – Nick Corona
    Dec 11, 2020 at 1:55
  • $\begingroup$ Yes, that is correct. $\endgroup$
    – Demetri Pananos
    Dec 11, 2020 at 2:04
  • $\begingroup$ Okay, that is essentially what I do. It's good to know I'm not going crazy. Sometimes it feels like it in industry because people have conflicting opinions that are hard to challenge due to hierarchy. $\endgroup$
    – Nick Corona
    Dec 11, 2020 at 2:06
  • $\begingroup$ @NickCorona Its hard to argue for "being right" in industry when "being wrong" comes with little consequence. At the end of the day, the bias imparted by your colleague's procedure likely won't cost your company thousand's of dollars. $\endgroup$
    – Demetri Pananos
    Dec 11, 2020 at 2:17
  • $\begingroup$ Oh, it could. These models affect millions of dollars. It's just hard to measure the effect, so your comment about being wrong coming with little consequence is apt. $\endgroup$
    – Nick Corona
    Dec 11, 2020 at 2:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.