2
$\begingroup$

I am wondering the following question. Probably it is a non-sense one but hope not too much..

Assume I have a binary classification model to build and I use a linear classifier like Logistic regression with L1 penalty (so the decision boundary is still linear) for feature selection.

Then I go through the training phase and I test several algorithms (linear and nonlinear classifier) for comparison. If I see that the best performance is given by a nonlinear classifier, does my feature selection using a linear one make sense? It is likely that if I use a nonlinear classifier for feature selection I get a different subset.

So how to deal with this?

Thanks. Luigi

$\endgroup$
1
1
$\begingroup$

Not a nonsense question at all! "Is it likely that if I use a nonlinear classifier for feature selection I get a different subset." Yes, that is quite possible, however, I think this is still a perfectly reasonable approach. Presumably you are doing this to cut down on time, so that you can quickly train many models on a smaller feat set. If your use case is not highly sensitive to getting the absolute highest possible performance and you are okay with a pretty good model, you are done.

If not computationally/time limited and you want to squeeze out more accuracy, I would do the exact process you describe, but then at the very end plug all the features back into your final non-linear model and do some kind of stepwise feature selection, either using tree-based feat importance if you are using trees, or a model agnostic version like permutation importance. You might still be left with a similar feat set, or you might pick up some extra features that help the non-linear version, but not L1.

$\endgroup$
5
  • $\begingroup$ Thanks! probably I can add 2 more feature selection to the L1 logistic regression: one would be a random forest ranking the features via MDI and another always a random forest but ranking via permutation importance..I would then train the model with these different sets (if different) and evaluate which is the best by checking performance on the test set. does it sound good in your opinion? $\endgroup$ – Luigi87 Dec 29 '20 at 15:51
  • 1
    $\begingroup$ Yep exactly, although idk if i'd bother with both types of feature importance, but can't hurt to see how they compare. Whatever model/feature set combo has the best performance on the test set at the end of the day is what you go with. And the more experiments you try the better. Another thing you could try is multiple levels of top features. So try all the features RF, then just use the top 80% most important features, then the 60% most important (where the feat importance list is recreated from the 80% run), then 40%, then 20%. This is kinda like tuning the L1 reg parameter. $\endgroup$ – Paul Fornia Dec 29 '20 at 16:01
  • 1
    $\begingroup$ See the "backwards elimination" section of wrapper methods: analyticsvidhya.com/blog/2016/12/…. $\endgroup$ – Paul Fornia Dec 29 '20 at 16:03
  • $\begingroup$ thanks for the suggestion. I would say I have three different methods now I can try: L1-LG, RF with permutation or MDI, and wrapper methods. All quite different in nature I would say $\endgroup$ – Luigi87 Dec 29 '20 at 16:14
  • $\begingroup$ just a doubt came in my mind, is permutation importance suitable for time series data? since you have some temporal order to maintain and serial correlation, permuting each column for me seems to violate both.. $\endgroup$ – Luigi87 Jan 4 at 10:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.