I want to select important features of a given dataset which contains a lot of noisy features. My question is general: If I select features, by let's say Recursive feature elimination or L1 penalty using one algorithm, does that separate noise from important features once and for all? Basically, would it be right approach to train a new model with features selected from a different algorithm or same algorithm with different hyperparameters?

  • 2
    $\begingroup$ Whether or not a feature is informative depends on the model. Here's an example, comparing random forest and logistic regression on a toy problem: stats.stackexchange.com/questions/164048/… $\endgroup$
    – Sycorax
    Mar 6, 2022 at 19:17
  • $\begingroup$ At least once features are not independent, features cannot cleanly be partitioned into "important features" and "noise". For example, if feature X1 and X2 share the same information about the outcome, any of these is unimportant given the other, but one of them is needed (unless the same information is in another feature). Different approaches may handle such cases differently (and in reality information shared by various features is rather the rule than the exception). $\endgroup$ Mar 6, 2022 at 20:35
  • $\begingroup$ @Sycorax This is from scikit-learn's documentation: "Linear models penalized with the L1 norm have sparse solutions: many of their estimated coefficients are zero. When the goal is to reduce the dimensionality of the data to use with another classifier, they can be used along with SelectFromModel to select the non-zero coefficients. In particular, sparse estimators useful for this purpose are the Lasso for regression, and of LogisticRegression and LinearSVC for classification". Is this correct? $\endgroup$ Mar 11, 2022 at 11:24
  • $\begingroup$ @Sycorax Also this, from the same page: "we make use of a LinearSVC coupled with SelectFromModel to evaluate feature importances and select the most relevant features. Then, a RandomForestClassifier is trained on the transformed output, i.e. using only relevant features. You can perform similar operations with the other feature selection methods and also classifiers that provide a way to evaluate feature importances of course. " $\endgroup$ Mar 11, 2022 at 12:09
  • $\begingroup$ Nothing you've quoted in any way disproves the claim made in the linked thread. The core claim -- that you can go from a larger number of features to a smaller number -- is vacuously true. There's no guarantee the selected features are relevant. But using a linear model to select features when the data-generating process is nonlinear can be disastrous, as demonstrated in the link I shared. $\endgroup$
    – Sycorax
    Mar 11, 2022 at 18:02

1 Answer 1


Short Answer: Yes and no, depending on what you mean by 'important'.

Long Answer: Let's consider using features $X := (x_1, \dots, x_d) \in \mathcal{X}$ to predict target $y \in \mathcal{Y}$. For instance $\mathcal{Y} = \{0, 1\}$ for some binary classification problems and $\mathcal{Y} \subset \mathbb{R}$ for single-output regression problems.

There are three related but different notions you might want to consider.

  • Feature Importance: A score that quantifies how important a feature is to a specific model for generating predictions/decisions. One such score is the Mean Absolute SHAP values. As a reminder, SHAP values are Shapley values computed using as the characteristic function $S \to v(S)=f_S(x_S)$ the actual model decision made by a model trained with features in the coalition $S$. We take the absolute value because it can be negative but any deviation from $0$ is a sign of 'importance/significance' and we take the mean because it depends on specific feature values whereas we want a property/characteristic of a feature (not a specific value thereof).

  • Feature Usefulness: A score that quantifies the contribution of a feature to the performance of a specific model. One possible implementation is (a score proportional to) Shapley values using as the characteristic function $S \to v(S)$ the (cross-validated) performance of the specific model using features $x_S$ in the coalition.

  • Feature Potential: A score that quantifies the intrinsic worth of a feature in a set of features for predicting your target $y$. One possible implementation is (a score proportional to) Shapley values using as the characteristic function $S \to v(S)$ the mutual information $I(y; x_S)$ between the target $y$ and the set of features $x_S$ in the coalition. Note that the mutual information $I(y; x_S)$ is tightly related to the highest performance you may achieve consistently using features $x_S$ to predict $y$ (no matter what model you use). See https://arxiv.org/abs/2107.08066 for more details.

Feature importance and feature usefulness are (obviously) model-specific, while feature potential is not!

In general, because a feature is important does not make it useful! Feature importance has nothing to do with model performance; they don't even depend on the target $y$!

If you pick a random model, you can compute feature importance for any feature and some can be much bigger than others. It does not make features with high importance scores more useful than features with low importance score!

By the way, for this reason and many others, you should never use Recursive Feature Elimination for feature selection. See this blog post for more details on why you should not use RFE: https://blog.kxy.ai/why-you-should-stop-using-recursive-feature-elimination/.

In general, because a feature is useful for one model does not make it useful for another! In other words, using one model to select features that you will then use in another model can be disastrous as already noted by @Sycorax.

When selecting features, you want to keep features that are useful to the specific type of models you are training, not features that are important or have potential. Checkout this page for some robust feature selection alternatives in Python: https://blog.kxy.ai/tag/feature-selection/.

Finally, the only reason you could consider a feature noise is if its feature potential is 0.

When a feature has 0 feature potential, you can conclude that it cannot help you predict the target $y$, no matter which model you use, when it is used either by itself or in conjunction with a subset of other features in $X$.

For instance, when you use Shapley values with the mutual information as the characteristic function, the feature potential score of feature $x_i$ reads:

$$P_X(x_i) \propto \sum_{S} \left[I(y; x_S \cup \{x_i\})- I(y; x_S)\right] = \sum_{S} I(y; x_i | x_S),$$ where the sum is over all feature subsets of $X$ that do not contain $x_i$.

Note that $I(y; x_S \cup \{x_i\}) \geq I(y; x_S)$, so $P_X(x_i)=0$ if and only if $I(y; x_S \cup \{x_i\}) = I(y; x_S)$ for any possible subset of $X$.

In particular, if you take $S=\emptyset$ the empty set, you get $I(y; x_i)=0$, which implies $y$ and $x_i$ are statistically (unconditionally) independent (a.k.a you cannot predict $y$ using $x_i$ by itself).

Also, $I(y; x_i | x_S)=0$ for all subset $x_S$ of $X$ implies that, if a feature has zero potential, then it is statistically independent from the target $y$ conditional on any subset of other features in the candidate vector $X$.

From an ML/stats standpoint, for a feature $x_i$ to be worth using in a supervised learning problem, it ought to be statistical dependent to the target $y$ either unconditionally, or conditional on some other features you are using. If this is not the case, the feature has no potential (usefulness).

However, the notion of feature potential is tied to the candidate set $X$ you are using. $x_i$ might very well have $0$ potential with respect to a candidate set, but some potential with respect to another.

Example: If $P_X(x_i)=0$ and $I(y; x_i|x_{S^*}) > 0$ then $P_{X \cup S^*} = I(y; x_i|x_{S^*}) > 0$. In plain english, in this example, while $x_i$ cannot be helpful to predict $y$ when used either by itself or in conjunction with some features in vector $X$, you can however do better when predicting $y$ with feature $x_i$ and features $x_{S^*} \not\subset X$ compared to using only features $x_{S^*}$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.