2
$\begingroup$

I'm looking at this example for LASSO regression in R: http://machinelearningmastery.com/penalized-regression-in-r/.

It mentions "step size" and "best_step". And, the documentation here also mentions this idea of a "step".

I have never heard of a step size being applied to LASSO regression. Can someone please help me understand where this is coming from?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
2
$\begingroup$

The LASSO procedure, as implemented in R, produces a family of models parameterized by "penalty" or "λ". This, when plotted against the coefficients of your explanatory variables, goes from a model with all coefficients 0 (excepting the intercept if present), all the way to the unpenalized model Example plot.

Step size is how λ changes between each calculation of the model.

"Best" step is just labeling the chosen model from amongst the family of models. The example uses minimum RSS as the metric for best.

Improve this answer
$\endgroup$
5
  • $\begingroup$ thank you for your answer. I don't understand the plot above. I assume you got it from here? beta plotted against beta over a constant (max beta) should be a constant slope. I don't see where $\lambda is on this plot...sorry if I'm being annoying $\endgroup$
    – makansij
    Commented Jun 29, 2016 at 5:11
  • $\begingroup$ Also, if step size is the effectively the learning rate, then wouldn't it make sense that the step size always be as small as possible since this is basically just an optimization problem? $\endgroup$
    – makansij
    Commented Jun 29, 2016 at 5:29
  • $\begingroup$ @Sother it looks like that plot is a ridge regression path, not a Lasso path. $\endgroup$
    – shadowtalker
    Commented Jul 10, 2016 at 0:56
  • $\begingroup$ I have never heard of this use for the term "step size" in the context of LAR, nor is it a configurable parameter in the lars() function. The LAR algorithm doesn't have a configurable step size as far as I know. $\endgroup$
    – shadowtalker
    Commented Jul 10, 2016 at 0:58
  • $\begingroup$ I should clarify: LAR does make use of a "step size" but the step size is specified according to a fixed formula. See Efron et al., (2003), "Least Angle Regression", bottom of page 7. $\endgroup$
    – shadowtalker
    Commented Jul 10, 2016 at 1:05
2
+50
$\begingroup$

They're misusing the word "step" here: they mean to refer to a "step number," not a "step size." In fact, it says as much in the documentation you link to (emphasis mine):

Mode="step" means the s= argument indexes the lars step number, and the coefficients will be returned corresponding * to the values corresponding to step s.

As for what steps have to do with Lasso regression, it's because a stepwise algorithm called LAR (Least Angle Regression) is being used to trace out the Lasso solution path. However, the step size is specified according to a specific formula; you can find details in the pleasantly readable original paper, Efron et al., (2003), "Least Angle Regression".

Improve this answer
$\endgroup$
1
$\begingroup$

I suggest having a look at An Introduction to Statistical Learning book. There, on page 255, you can see that the argument "s" is just the value of $\lambda$. In the case of the example from the book, this value was chosen based on cross-validation. Logically, step size would be a coefficient used in the underlying optimisation algorithm implemented with the glmnet() function in R (think of it as something similar to $\gamma$ in gradient descent), so a bigger step size in lasso would mean that global optimum for $\lambda$ could be found faster but also that this optimum could be missed. Therefore, step size should be chosen wisely.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Yes, LASSO has a stepwise solution because it is a non-convex optimization problem. But, what would be so bad about setting the step size s small as possible, and then basically guarantee that you find the global minimum. $\endgroup$
    – makansij
    Commented Jul 18, 2016 at 1:35
  • $\begingroup$ Of course you can set it to very small one, it's just a tradeoff between optimisation accuracy (finding the global minimum) and computation time. $\endgroup$
    – slazien
    Commented Jul 18, 2016 at 9:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.