11
$\begingroup$

I am experimenting with stepwise regression for the sake of diversity in my approach to the problem. So, I have 2 questions:

  1. What are the advantages of stepwise regression? What are its specific strengths?

  2. What do you think about hybrid approach, where you use stepwise regression to select features, and then apply regular regression taking all the selected features together?

$\endgroup$
15
$\begingroup$

The primary advantage of stepwise regression is that it's computationally efficient. However, its performance is generally worse than alternative methods. The problem is that it's too greedy. By making a hard selection on the the next regressor and 'freezing' the weight, it makes choices that are locally optimal at each step, but suboptimal in general. And, it can't go back to revise its past choices.

As far as I'm aware, stepwise regression has generally fallen out of favor compared to $l_1$ regularized regression (LASSO), which tends to produce better solutions.

Tibshirani (1996). Regression Shrinkage and Selection via the Lasso

LASSO penalizes the $l_1$ norm of the weights, which induces sparsity in the solution (many weights are forced to zero). This performs variable selection (the 'relevant' variables are allowed to have nonzero weights). The degree of sparsity is controlled by the penality term, and some procedure must be used to select it (cross-validation is a common choice). LASSO is more computationally intensive than stepwise regression, but a number of efficient algorithms exist. Some examples are least angle regression (LARS), and an approach based on coordinate descent.

A similar approach to what you suggested in (2) is called orthogonal matching pursuit. It's a generalization of matching pursuit, which is the name for stepwise regression in the signal processing literature.

Pati et al. (1993). Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition

On each iteration, the next best regressor is added to the active set. Then, the weights for all regressors in the active set are recomputed. Because of the reweighting step, this approach is less greedy (and has better performance) than the regular matching pursuit/stepwise regression. But, it still employs a greedy search heuristic.

All of these approaches (stepwise regression, LASSO, and orthogonal matching pursuit) can be thought of as approximations of the following problem:

$$\underset{w}{\min} \| y - X w \|_2^2 \quad \text{s.t. } \|w\|_0 \le c$$

In a regression context, columns of $X$ correspond to the independent variables and $y$ to the dependent variable. In signal processing, columns of $X$ correspond to basis functions and $y$ is a signal to approximate. The goal is to find a sparse set of weights $w$ that give the best (least squares) approximation of $y$. The $l_0$ norm simply counts the number of non-zero entries in $w$. Unfortunately, this problem is NP-hard, so approximation algorithms must be used in practice. Stepwise regression and orthogonal matching pursuit attempt to solve the problem using a greedy search strategy. LASSO reformulates the problem using a relaxation of the $l_0$ norm to the $l_1$ norm. Here, the optimization problem becomes convex (and thus tractable). And, although the problem is no longer identical, the solution is similar. If I recall correctly, both LASSO and orthogonal matching pursuit have been proved to recover the exact solution under certain conditions.

$\endgroup$
8
$\begingroup$

Stepwise selection is not generally a good idea. To understand why, it may help you to read my answer here: Algorithms for automatic model selection.

As far as advantages go, in the days when searching through all possible combinations of features was too computationally intensive for computers to handle, stepwise selection saved time and was tractable. However, note that the problems discussed in my linked answer above apply just as much to 'best subset' regression, so stepwise does not produce a good solution, just a bad solution faster.

Your idea of a hybrid approach would be fine, so long as the second model (with the selected features) was fitted on a new dataset.

$\endgroup$
  • $\begingroup$ Regarding what the OP called a "hybrid approach" (not quite sure why it's hybrid), you mean it's fine in the sense that the estimates of the coefficients of the model on the second new datasets should be fine (while biased and problematic on the original data), as long as the new dataset is large enough? It would of course potentially be a poor model, because it was selected in a bad way on the first dataset, simply it's coefficients would be estimated in a less problematic dataset. $\endgroup$ – Björn Jun 10 '16 at 5:32
  • $\begingroup$ Plus it's still often impossible to look through all possible combinations, because the number of different variables we have data on grows even faster than computing power, and people have more and more ideas about what to include in their models. $\endgroup$ – Stephan Kolassa Jun 10 '16 at 7:08
1
$\begingroup$

I have just taked a google search for what is Stepwise regression. I'm not sure if I fully understand it, but here is my first thought

  • It's greedy so it can not produce the good solution as Lasso does. I prefer Lasso
  • It's simple, easy to use, easy to code
  • After you use Stepwise regression, you already end up with a trained model that use selected features, so you do not need to use an other regression step as you mentioned as hybrid approach
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.