Suppose that I am interested in applying Recursive Feature Elimination (such as
sklearn), and my data set is of medium size, with $n>p$, where $p$ is the number of features. Let's say I have reason to believe that the data is relatively noisy, and I believe that feature selection may improve classification accuracy.
If I wanted to implement RFE with a support vector classifier (say
SVC() with a linear kernal), the model must be trained, then passed to
RFE(), which then can find a subset of the features to use in the final model. Then, do we retrain the SVC on the dataset using the subset of the features? So my question:
Is this the correct interpretation? If so, then how do we address the tuning of hyperparameters within the model? (Linear) SVC's use the parameter $C$ to control the amount of regularization which can drastically affect the bias/variance tradeoff. Thus, do we optimize $C$ first, then pass the model to
RFE(), derive best subset feature selection, then fit the same model on the subsetted features? Or do we choose an arbitrary $C$ (say 1), pass to
RFE(), then train and tune a final model on the remaining features? Of course, all tuning is being done with cross-validation in this scenario.
The former option seems more correct to me, but I also think that we are missing out on some possible improvements in accuracy by not tuning the model again on the subset of the features. (Perhaps two tunings? Before RFE and after RFE?)
Can we generalize this procedure to other classifiers? What about if an $l_1$ penalty is available?