In my ML pipeline, I normally perform feature selection, by performing a few of the tests mentioned below, the ones relevant to the model. I tend to drop features with negative outlier to the rest of the features, but I wonder whether more data is better, and leave the feature selection to the algorithm to dechiper which features to prioritise.

  1. Is identifying and removing extremely multicollinear regression variables always in benefit of the performance of such a model?
  2. Is identifying features in a classification model which do no depend on the label set via a Chi-squares test for independence always in benefit of the performance of such a model?
  3. The features_importances function in sklearn measures the average gain of purity by splits of a given variable. In the cases where the importance level is extremely low, it is always advisable to drop such features.

1 Answer 1


Feature selection often makes generalisation performance worse rather than better, so I would recommend against routinely including it in the analysis unless identifying the relevant attributes is a specific aim.

Regularisation (which is used in most modern ML algorithms) is able to deal with colinearity.

Testing individual features for independence is not a good idea as some features may be useless on their own, but highly informative in combination with others. Consider a 2-d dataset where one class consists of a spherical blob at the origin and the other class consists of a doughnut centered on the origin. In that case, you may be able to get perfect separation with both features, but a chi-squared test will probably reject both features considered individually.

  • $\begingroup$ Is there any cases where one should chose to drop a feature, apart from the obvious case where the values of the feature is constant? $\endgroup$ Commented Dec 30, 2021 at 12:58
  • $\begingroup$ I might drop them based on advice from a domain expert who understands the data. It may be worth using a machine learning method that can down-weight uninformative attributes (e.g. L1 regularisation) so that the ML method itself can identify them, rather than deleting them first. $\endgroup$ Commented Dec 30, 2021 at 14:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.