Are important features or noise model agnostic?

Question

I want to select important features of a given dataset which contains a lot of noisy features. My question is general: If I select features, by let's say Recursive feature elimination or L1 penalty using one algorithm, does that separate noise from important features once and for all? Basically, would it be right approach to train a new model with features selected from a different algorithm or same algorithm with different hyperparameters?

Answer 1

Short Answer: Yes and no, depending on what you mean by 'important'.

Long Answer: Let's consider using features $X := (x_1, \dots, x_d) \in \mathcal{X}$ to predict target $y \in \mathcal{Y}$. For instance $\mathcal{Y} = \{0, 1\}$ for some binary classification problems and $\mathcal{Y} \subset \mathbb{R}$ for single-output regression problems.

There are three related but different notions you might want to consider.

• Feature Importance: A score that quantifies how important a feature is to a specific model for generating predictions/decisions. One such score is the Mean Absolute SHAP values. As a reminder, SHAP values are Shapley values computed using as the characteristic function $S \to v(S)=f_S(x_S)$ the actual model decision made by a model trained with features in the coalition $S$. We take the absolute value because it can be negative but any deviation from $0$ is a sign of 'importance/significance' and we take the mean because it depends on specific feature values whereas we want a property/characteristic of a feature (not a specific value thereof).

• Feature Usefulness: A score that quantifies the contribution of a feature to the performance of a specific model. One possible implementation is (a score proportional to) Shapley values using as the characteristic function $S \to v(S)$ the (cross-validated) performance of the specific model using features $x_S$ in the coalition.

• Feature Potential: A score that quantifies the intrinsic worth of a feature in a set of features for predicting your target $y$. One possible implementation is (a score proportional to) Shapley values using as the characteristic function $S \to v(S)$ the mutual information $I(y; x_S)$ between the target $y$ and the set of features $x_S$ in the coalition. Note that the mutual information $I(y; x_S)$ is tightly related to the highest performance you may achieve consistently using features $x_S$ to predict $y$ (no matter what model you use). See https://arxiv.org/abs/2107.08066 for more details.

Feature importance and feature usefulness are (obviously) model-specific, while feature potential is not!

In general, because a feature is important does not make it useful! Feature importance has nothing to do with model performance; they don't even depend on the target $y$!

If you pick a random model, you can compute feature importance for any feature and some can be much bigger than others. It does not make features with high importance scores more useful than features with low importance score!

By the way, for this reason and many others, you should never use Recursive Feature Elimination for feature selection. See this blog post for more details on why you should not use RFE: https://blog.kxy.ai/why-you-should-stop-using-recursive-feature-elimination/.

In general, because a feature is useful for one model does not make it useful for another! In other words, using one model to select features that you will then use in another model can be disastrous as already noted by @Sycorax.

When selecting features, you want to keep features that are useful to the specific type of models you are training, not features that are important or have potential. Checkout this page for some robust feature selection alternatives in Python: https://blog.kxy.ai/tag/feature-selection/.

Finally, the only reason you could consider a feature noise is if its feature potential is 0.

When a feature has 0 feature potential, you can conclude that it cannot help you predict the target $y$, no matter which model you use, when it is used either by itself or in conjunction with a subset of other features in $X$.

For instance, when you use Shapley values with the mutual information as the characteristic function, the feature potential score of feature $x_i$ reads:

$$P_X(x_i) \propto \sum_{S} \left[I(y; x_S \cup \{x_i\})- I(y; x_S)\right] = \sum_{S} I(y; x_i | x_S),$$ where the sum is over all feature subsets of $X$ that do not contain $x_i$.

Note that $I(y; x_S \cup \{x_i\}) \geq I(y; x_S)$, so $P_X(x_i)=0$ if and only if $I(y; x_S \cup \{x_i\}) = I(y; x_S)$ for any possible subset of $X$.

In particular, if you take $S=\emptyset$ the empty set, you get $I(y; x_i)=0$, which implies $y$ and $x_i$ are statistically (unconditionally) independent (a.k.a you cannot predict $y$ using $x_i$ by itself).

Also, $I(y; x_i | x_S)=0$ for all subset $x_S$ of $X$ implies that, if a feature has zero potential, then it is statistically independent from the target $y$ conditional on any subset of other features in the candidate vector $X$.

From an ML/stats standpoint, for a feature $x_i$ to be worth using in a supervised learning problem, it ought to be statistical dependent to the target $y$ either unconditionally, or conditional on some other features you are using. If this is not the case, the feature has no potential (usefulness).

However, the notion of feature potential is tied to the candidate set $X$ you are using. $x_i$ might very well have $0$ potential with respect to a candidate set, but some potential with respect to another.

Example: If $P_X(x_i)=0$ and $I(y; x_i|x_{S^*}) > 0$ then $P_{X \cup S^*} = I(y; x_i|x_{S^*}) > 0$. In plain english, in this example, while $x_i$ cannot be helpful to predict $y$ when used either by itself or in conjunction with some features in vector $X$, you can however do better when predicting $y$ with feature $x_i$ and features $x_{S^*} \not\subset X$ compared to using only features $x_{S^*}$.