Why do we need Forward Selection, Forward Stagewise and LARS for solve linear regression

Question

Forward Selection, Forward Stagewise and LARS are all used to solve the regression problem: $$y = \beta^Tx.$$ I understand how they work, but don't know the motivation of those algorithms. Since we already have the gradient descent or SVD to solve the least square estimation.

Are they just used to simplify the complexity? Since at most $n = dimension$ iterations is needed compared with gradient descent. Here we ignore the modified LARS for solving LASSO.

Answer 1

Those algorithms provide different ways to add predictors sequentially to a regression model, rather than solving the model all at once with standard least-squares fitting. They are used when the standard least-squares solution doesn't exist or you perform automated variable selection.

If the number of predictors in the model exceeds the number of observations, then there is no unique least-squares solution. These algorithms can select a subset of predictors to generate a solution.

Even when there is a unique least-squares solution, these algorithms can be used with a stopping rule to select a subset of predictors. Such an automated approach is often not wise. For example, you will typically find that the specific predictors returned by these algorithms differ among re-samples of the data. Thus any decision based on such results about which are the "most important" predictors is usually on shaky grounds.

With care, these approaches can sometimes be used for prediction and inference. The Statistical Learning with Sparsity book by Hastie et al goes into detail about their proper use.