# What is "step size" in LASSO regression?

I'm looking at this example for LASSO regression in R: http://machinelearningmastery.com/penalized-regression-in-r/.

It mentions "step size" and "best_step". And, the documentation here also mentions this idea of a "step".

I have never heard of a step size being applied to LASSO regression. Can someone please help me understand where this is coming from?

• thank you for your answer. I don't understand the plot above. I assume you got it from here? beta plotted against beta over a constant (max beta) should be a constant slope. I don't see where $\lambda is on this plot...sorry if I'm being annoying Jun 29, 2016 at 5:11 • Also, if step size is the effectively the learning rate, then wouldn't it make sense that the step size always be as small as possible since this is basically just an optimization problem? Jun 29, 2016 at 5:29 • @Sother it looks like that plot is a ridge regression path, not a Lasso path. Jul 10, 2016 at 0:56 • I have never heard of this use for the term "step size" in the context of LAR, nor is it a configurable parameter in the lars() function. The LAR algorithm doesn't have a configurable step size as far as I know. Jul 10, 2016 at 0:58 • I should clarify: LAR does make use of a "step size" but the step size is specified according to a fixed formula. See Efron et al., (2003), "Least Angle Regression", bottom of page 7. Jul 10, 2016 at 1:05 They're misusing the word "step" here: they mean to refer to a "step number," not a "step size." In fact, it says as much in the documentation you link to (emphasis mine): Mode="step" means the s= argument indexes the lars step number, and the coefficients will be returned corresponding * to the values corresponding to step s. As for what steps have to do with Lasso regression, it's because a stepwise algorithm called LAR (Least Angle Regression) is being used to trace out the Lasso solution path. However, the step size is specified according to a specific formula; you can find details in the pleasantly readable original paper, Efron et al., (2003), "Least Angle Regression". I suggest having a look at An Introduction to Statistical Learning book. There, on page 255, you can see that the argument "s" is just the value of$\lambda$. In the case of the example from the book, this value was chosen based on cross-validation. Logically, step size would be a coefficient used in the underlying optimisation algorithm implemented with the glmnet() function in R (think of it as something similar to$\gamma$in gradient descent), so a bigger step size in lasso would mean that global optimum for$\lambda\$ could be found faster but also that this optimum could be missed. Therefore, step size should be chosen wisely.
• Yes, LASSO has a stepwise solution because it is a non-convex optimization problem. But, what would be so bad about setting the step size s small as possible, and then basically guarantee that you find the global minimum. Jul 18, 2016 at 1:35