Differences: between Forward/Backward/Bidirectional || Stepwise/Sequential || Selection/Regression?

Question

I am dealing with wrapper methods for feature selection. Thereby I came across the terms: Forward/Backward/Bidirectional, Stepwise/Sequential and Selection/Regression. These different terms confuse me.

I understand it as follows:

• Stepwise and Sequential are synonyms. These terms describe the iterative approach of the methods.
• Forward methods start with a null model or no features from the entire feature set and select the feature that performs best according to some criterion (t-test, partial F-test, strongest minimization of MSE, etc.)
• Backward methods start with the entire feature set and eliminate the feature that performs worst according to the above criteria.
• Bidirectional methods are a mixture of the methods described above.
• Selection and regression are sometimes used as synonyms. This seems to me to have grown historically, because linear regression used to be used as a forward method.

But then there are the following differences:

• The term stepwise can be understood in a narrower sense. According to this method, if a variable was included in the forward selection, it is checked whether the variables already present in the model are still significant. If this is not the case for a variable, it is removed from the model. It can be included in the set of variables to be selected.
• And then the procedures can be combined with a cross-validation to counteract an overfit. This represents a modification of the "vanilla" forward regression practiced in the past.

Answer 1

Stepwise feature selection is a "greedy" algorithm for finding a subset of features that optimizes some arbitrary criterion. Forward, backward, or bidirectional selection are just variants of the same idea to add/remove just one feature per step that changes the criterion most (thus "greedy").

Greedy algorithms are often employed where finding a global maximum is computationally infeasible, in the hope that the result is good enough.

Stepwise selection can be applied to any prediction algorithm (so-called "wrapper methods") and even without a prediction algorithm at all (so-called "filter methods"). All you need is a function that computes the criterion for a given data set and feature subset selection.

Concerning your last two questions:

1. Statistical significance only makes sence in the context of statistical models, which is only a (minor) subset of machine learning algorithms. Even if you use a model-based approach, p-values after variable selection are biased and thereby of only limited use.
2. To detect overfit, you would need an outer additional cross-validation loop, because most criteria for feature slection already minimize the cross-vaidated error rate (e.g. minimizing AIC is asymptotically equivalent to minimizing the leave-one-out error rate).

It should be noted that Monte Carlo methods like Boruta or Genetic Algorithms also address the feasibility problem and do not suffer from getting stuck in a local optimum. And if you use neural nets, feature selection is built into the training process ("deep features"). There are methods specifically tailored for linear regression (e.g. LASSO), but stepwise selection is not restricted to linear regression.