What is the main purpose of Feature Selection?

Question

I have a small medical dataset (200 samples) that contains only 6 cases of the condition I am trying to predict using machine learning. So far, the dataset is not proving useful for predicting the target variable and is resulting in models with 0% recall and precision, probably due to the scarcity of the minority class.

However, in order to learn from the dataset, I applied Feature Selection techniques to deduct what features are useful in predicting the target variable and see if this supports or contradicts previous literature on the matter.

When I reran my models using the reduced dataset, this still resulted in 0% recall and precision. So the prediction performance has not improved using feature selection. But the features returned by the applying Feature Selection have given me more insight into the data.

So my question is, is the purpose of Feature Selection:

• to improve prediction performance
• or can the purpose be identifying relevant features in the prediction and learning more about the dataset

So in other words, is Feature Selection just a tool to achieve improved performance, or can it be an end in itself?

Answer 1

In short, both answers are correct.

Feature selection has two main purposes:

• It reduces the number of features in the dataset. This reduces the model training time and reduces the chance of overfitting.
• It helps you understand the data i.e. which features in the dataset are the most important.

Hence, I would not expect feature selection to help when training your model, unless you are overfitting the training data.

Answer 2

Allow me to be a contrarian and say that feature selection is overrated. My post here discusses feature selection when features are correlated, but the same bias-variance argument applies to uncorrelated features. Briefly, by excluding features, you are forcing their coefficients to be zero, and even if this results in lower variance, you might incur enough of a bias (by forcing the estimate of a nonzero parameter to be zero) that you are worse-off for having done the feature selection.

Further, feature selection tends to be unstable. Our Frank Harrell, founding chair of the Department of Biostatistics at Vanderbilt University, has written about this. (That Regression Modeling Strategies book he mentions should be considered mandatory reading for anyone doing predictive modeling (which includes all supervised machine learning).) If you want to get into specific relationships, particularly causal relationships, that requires much more delicate handling than predictive modeling requires. As Harrell mentions in the link, selecting a feature is related to overestimating its effect; conversely, electing to omit a feature is related to underestiating its effect.

Note that your desire to do feature selection is driven by your use of an improper scoring rule. You are correct that your class imbalance is likely driving you not to predict the minority class, but this is probably correct behavior. The minority class might always be unlikely. By using improper scoring rules, you are missing this fact and missing out on being able to predict the probability of being in your minority class. Classification requires a threshold, which your software default is likely to have set at $0.5$, even though the minority class might never truly have a probability of $0.5$ under any conditions (combination of features). The simplest way to deal with this is to adjust the threshold. A more advanced approach would deal directly with predicted probabilities. Again, Frank Harrell has written about this in two great blog posts [1, 2].

Finally, with only six cases of the minority condition, it is unlikely that you have enough data to do much at all, even if you use best practices, and it is important to be able to relay to stakeholders when they are pursuing a hopeless problem or using a hopeless approach.

Answer 3

I will add another motivation: it helps the model with learning more from small datasets (which is one aspect of overfitting). In an ML task where you do not have a stretchable budget for more data points, feature selection can be one of your best tools.

Answer 4

What is the main purpose of Feature Selection?

Feature selection is probably the most important topic in model building. Indeed asking clarification about the purpose of variables selection looks like asking clarification about the purpose of the model itself.

In my view most models, for example most regression models, can have two main goals: Causal inference and prediction. See here: What is the relationship between minimizing prediction error versus parameter estimation error? and links therein.

Therefore

So my question is, is the purpose of Feature Selection:

• to improve prediction performance

this is surely a right purpose for features selection. In particular it is the purpose if the model is built for prediction.

So my question is, is the purpose of Feature Selection:

• or can the purpose be identifying relevant features in the prediction and learning more about the dataset

this sentences is a bit tautological "the purpose of features selection is ... the purpose of identifying relevant features".

Moreover your point sound like a conflation between prediction and something like explanation. This is not a good idea, read here: Minimizing bias in explanatory modeling, why? (Galit Shmueli's "To Explain or to Predict")

The second scope for variables selection is related with causal inference. In short this purpose is to avoid endogeneity and related distortions in estimation of causal parameters of interest. In other words, variables selection is useful for identification.

See here for example: Which OLS assumptions are colliders violating?