# Understanding 'aggressiveness' of lasso, forward stepwise selection and best subset selection in Hastie, T., Tibshirani, R. & Tibshirani, R.J. (2017)

Hastie et al. (2017) explain how the above mentioned methods perform depending on the signal-to-noise ratio (SNR) with their varying 'aggressiveness'. Now I don't understand why the different methods vary in their aggressiveness (meaning the number of predictors they include in the final model?) and how this relates to the SNR. I think I understand bias-variance trade-off and how it relates to better performance of lasso in some scenarios but the authors give additional explanations that I don't get.

In their explanation the authors write that

"the fitted values from the lasso (for any fixed $$\lambda \geq 0$$) are continuous functions of y (Zou et al., 2007; Tibshirani and Taylor, 2012), whereas the fitted values from forward stepwise and best subset selection (for fixed $$k \geq 1$$) jump discontinuously as y moves across a decision boundary for the active set" (p. 3)

Could somebody clarify for me what the 'decision boundary' is and what is meant by the active set (the set of predictors selected?). The authors also relate aggressiveness to the degrees of freedom, a point I can't grasp.

I'd appreciate an intuitive explanation in addition to any equations because I don't have a strong math background.

Hastie, T., Tibshirani, R., & Tibshirani, R. J. (2017). Extended Comparisons of Best Subset Selection, Forward Stepwise Selection, and the Lasso. ArXiv:1707.08692 [Stat]. http://arxiv.org/abs/1707.08692

• Might the active set be the regressors that belong to the data generating process rather than ones selected in a model? Aug 17, 2020 at 14:13

From its use in the linked paper, the "active set" is the set of predictors that are being added to the model as it is being built. See the initial use of the phrase with respect to forward stepwise, in which you start with an empty "active set" and add predictors to the set sequentially.

Say that this is a linear regression model, so your criterion for deciding on the "best" model involves the mean-square difference between the observed values of the outcomes $$y$$ and their predicted values $$\hat y$$. The issue is how might noise in the observed values of $$y$$ pose difficulties for making predictions from the "best" model that's chosen based on the observed data.

Say you fit by forward-stepwise or best-subset, and random noise in your set of observed values $$y$$ means that your mean-squared error criterion pushes the choice of the "best" model from a 3-predictor to a 4-predictor model. That's crossing a decision boundary. As there's a whole new predictor being added, the predicted values $$\hat y$$ for any set of predictor values will differ by jumps between the two models, so later predictions might depend heavily on the noise in the original observations. You can think about this as a risk of these approaches potentially trying to fit noise in a particular data sample.

With lasso, you aren't just adjusting the number of predictors as you change the penalty value $$\lambda$$. You are also adjusting the penalization of the corresponding regression-coefficient magnitudes. So any random noise in the observations of $$y$$ will lead to continuous rather than stepwise changes in the ultimate predictions $$\hat y$$ made by the model. On that basis lasso can be considered less "aggressive" in its modeling, as its ultimate predictions tend not to overfit noise in the original data.

From ISLR, page 35 (with $$\hat f$$ representing the predicted value), describing the bias-variance tradeoff:

Variance refers to the amount by which $$\hat f$$ would change if we estimated it using a different training data set.

That's what the above argument is about. A slight change of noise in a training set can make a large difference in the predictions from a model devised by stepwise or best-subset methods. The penalization inherent in lasso minimizes the variance in that sense of the word.

Whether stepwise and best-subset methods are associated with more "instability" might depend on your definition of that term. If by "instability" you mean differences in the final set of predictors selected as you move from training set to training set, all predictor-selection methods including lasso have that instability. Try modeling on repeated bootstrap samples from a data set to illustrate that type of instability.

On the other hand, with the same size of training data, the larger numbers of effective degrees of freedom used up by stepwise and best-subset methods make them more prone to overfitting than lasso. That overfitting is pretty much included in the use of the word "variance" above, so if by "instability" you mean high "variance" then yes, that's the case. Even if lasso models trained on different training sets differ in terms of the predictors maintained, they are less likely to differ in terms of predictions.

Finally, the larger number of degrees of freedom means that p-values naively calculated for stepwise and best-subset models aren't reliable. They don't take into account the use of the data to define the model.

• Thanks a lot! Very comprehensible explanation of decision boundary and the jumps of $\hat{y}$. Could you relate this to the bias-variance tradeoff? And I've been thinking about the role of degrees of freedom - does the greater number of degrees of freedom for best subset and forward stepwise mean that they are more instable than lasso? If you could explain this in your answer I can accept it.
– Robn
Aug 18, 2020 at 8:32