I am currently using feature selection approaches like filter, wrapper, embedded etc.

All these methods give different set of features and I rank them based on their frequency of occurrence in other feature selection approach.

Ex: If Age occurs in all 6 feature selection algorithm, then they will have a rank as 6. If gender occurs only in 2 feature selection algorithm, its rank would be 2.

So I arrange them in descending order and choose the features which has occurred in at least 3-4 feature selection algorithm.

But where I am trying to seek your help is

1) Is there any systematic way where I get only a subset of features which returns best output? I thought genetic algorithm for feature selection will return a feature subset which will give high output. But unfortunately it lists all features with their importance. I don't wish to define a threshold myself to select few. Feel that may not be the best approach.

2) Is there any algorithm, like RFE but which does exhaustive search like genetic algorithm and finally returns us the best feature set? Mix of RFE (for best feature set) and genetic algorithm (exhaustive search) is what I need

3) I wish to have an exhaustive search in feature space and finally provide me the best feature set which will provide best f1-SCORE which is my objective.

I am looking for a systematic way/algorithmic way to do this rather than me deciding to pick features which occurs at least in 3 feat selection algorithms/user defined thresholds may not be best ones

Hope my question is clear. Can you help me with this?


1 Answer 1


Actually, Genetic Algorithm based approaches, can also give you the optimum list of features besides giving you the ranking. I suspect that the specific library you are using is simply giving you the feature ranking.

I would suggest to use an Optimization algorithm and code up the objective function(rmse or r2) yourself and perform optimization. The optimized result will give you the final list of features(without needing to rank) for which the rmse was minimum.

Here are some python packages and the optimization algorithms implemented in them:
Particle Swarm: https://pythonhosted.org/pyswarm/

Differential Evolution: https://docs.scipy.org/doc/scipy-0.17.0/reference/generated/scipy.optimize.differential_evolution.html

  • $\begingroup$ Thanks. Upvoted. Quick question So even if genetic algorithm uses a tree based models, you say it will just give a best performing feature subset. Am I right to understand this way? $\endgroup$
    – The Great
    Commented Jan 3, 2020 at 12:55
  • $\begingroup$ Genetic algorithms work equally well for tree based models and for polynomial models. Actually, the best feature subset is the direct result of optimization. The feature importance is a derived result. $\endgroup$ Commented Jan 3, 2020 at 12:57
  • $\begingroup$ Is there any package that does this by default? I will not be able to code it. So is there any specific package that you know of? $\endgroup$
    – The Great
    Commented Jan 3, 2020 at 12:57
  • $\begingroup$ I generally manually code up so I have not used packages directly; However, Following are some packages that I know of that provide a good interface: github.com/kaushalshetty/FeatureSelectionGA and github.com/manuel-calzolari/sklearn-genetic $\endgroup$ Commented Jan 3, 2020 at 13:02
  • $\begingroup$ The ones that I mentioned above provide a simple interface to feature selection. $\endgroup$ Commented Jan 3, 2020 at 13:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.