Use L1 logistic regression for feature selection but a nonlinear classifier for the training. Is that feature selection reliable? I am wondering the following question. Probably it is a non-sense one but hope not too much..
Assume I have a binary classification model to build and I use a linear classifier like Logistic regression with L1 penalty (so the decision boundary is still linear) for feature selection.
Then I go through the training phase and I test several algorithms (linear and nonlinear classifier) for comparison.
If I see that the best performance is given by a nonlinear classifier, does my feature selection using a linear one make sense? It is likely that if I use a nonlinear classifier for feature selection I get a different subset.
So how to deal with this?
Thanks. Luigi
 A: Not a nonsense question at all! "Is it likely that if I use a nonlinear classifier for feature selection I get a different subset." Yes, that is quite possible, however, I think this is still a perfectly reasonable approach. Presumably you are doing this to cut down on time, so that you can quickly train many models on a smaller feat set. If your use case is not highly sensitive to getting the absolute highest possible performance and you are okay with a pretty good model, you are done.
If not computationally/time limited and you want to squeeze out more accuracy, I would do the exact process you describe, but then at the very end plug all the features back into your final non-linear model and do some kind of stepwise feature selection, either using tree-based feat importance if you are using trees, or a model agnostic version like permutation importance. You might still be left with a similar feat set, or you might pick up some extra features that help the non-linear version, but not L1.
