I am wondering the following question. Probably it is a non-sense one but hope not too much..
Assume I have a binary classification model to build and I use a linear classifier like Logistic regression with L1 penalty (so the decision boundary is still linear) for feature selection.
Then I go through the training phase and I test several algorithms (linear and nonlinear classifier) for comparison. If I see that the best performance is given by a nonlinear classifier, does my feature selection using a linear one make sense? It is likely that if I use a nonlinear classifier for feature selection I get a different subset.
So how to deal with this?