In lasso, we first standardize all variables to mean=0, var=1. As such, a beta is simply correlation, right? We want to keep the sum of absolute values of betas below certain value, so what do we do:

we keep adding variables, highest correlation first(in absolute value), until the sum of absolute values of their betas, i.e. of their correlations, is below the threshold.

Is this correct? If not, what's the correct explanation of lasso in terms of correlations(vs betas)?

  • 2
    $\begingroup$ In multiple regression, the betas for standardized variables are not quite the same as correlations (see: Are standardized betas in multiple linear regression partial correlations?). $\endgroup$ – gung Dec 15 '15 at 17:26
  • $\begingroup$ I mean, the betas before returning to the original space, i.e. before multiplying with the original stdevs that we divided by in the first place. $\endgroup$ – LassoUser Dec 15 '15 at 17:28

First, note the point that @gung made in a comment: betas for standardized variables are not quite the same as correlations. This issue holds true even "before returning to the original space," as you put it, as the answer from @glen_b on that page explains.

Your suggestion about a strict relation between correlations and the corresponding standardized regression coefficients would not (necessarily) hold, however, even in an application of the LASSO algorithm with a single predictor variable.

LASSO minimizes the residual sum of squares subject to a penalty on the sum of the magnitudes of the coefficients. With any non-zero penalty, the regression coefficient returned by LASSO will thus be smaller in magnitude than the standard linear regression coefficient. So LASSO is not simply adding one variable at a time at its "full" regression coefficient.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.