Good day,

I had a question regarding feature selection. I found a very handy $\texttt{python}$ library $\texttt{mlxtend}$ which has a Stepwise feature selection function $\texttt{sfs}$. For an assignment I am doing, I want to fit and compare a couple of models for both classification and regression.

My question is, do I perform this once on a random model and keep the same features throughout (e.g. for classification just use the features proposed by a logistic regression model) and then fit all the other models on the same features? Or do I perform feature selection for each new model?

Just picking one set of covariates feels the most correct and I did not question it before but I am a little stumped now. Are two models "comparable" if they use different covariates?


1 Answer 1


So if(!) your goal is model comparison, then each model should be given the best chance to possibly "win" the competition and thus feature selection should be done separately for each tested model. This is because for making predictions different models may rely on different features.

Also, even if you constrain yourself to a single model (or rather model category such as SVMs), you may still define different actual models for comparison by defining a SVM for each possible combination of input features. So, for example, if your data set entails three features and your only model category is SVM, then an actual model that is eligible for comparison may still be defined for each of the $2^3$ possible subsets of features.

This is because a model may be defined for each of the subsets by introducing (fixed) feature weight vectors $\mathbf{w} \in \{0,1\}^3$ put in front of the same general SVM. As these weights are clearly part of a models parameter set, assigning different values to them will yield different models.

Hope this helps...

  • $\begingroup$ Could you direct me to some literature that supports this? $\endgroup$ Apr 13, 2020 at 14:30
  • $\begingroup$ I can only recommend the standard literature on model comparison, such as "Bishop: Pattern Recognition and Machine Learning" and "Murphy: Machine Learning: A Probabilistic Perspective", from which it becomes clear that the same model (paradigm) applied to different feature subsets yields indeed different models eligible to model comparison $\endgroup$ Apr 13, 2020 at 16:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.