2
$\begingroup$

In lasso regression, increasing the regularisation strength/shrinkage penalty eventually forces all of the regression coefficients to zero. In this instance the regression is logistic.

The plot shows the 'trajectory' of each coefficient as the regularisation strength increases along the x axis. The explanatory variables have been standardised.

Values of regression coefficients as shrinkage penalty increases

Coefficients 3 and 7 (7 is the upper green curve whose label is slightly obscured) for example 'survive' the longest, i.e. take the largest penalty values to force them to zero. Other coefficients, like 9, quickly die off to zero for smaller penalty values so don't affect the response for large enough penalty values.

(1) Can one interpret this as coefficients 3 and 7 are most important in terms of association with the response?

(2) What determines the rate of decay of the coefficients with shrinkage penalty? Why do some coefficients become zero quickly, and some remain non-zero longer?

$\endgroup$
  • 3
    $\begingroup$ 1. Yes. 2. the shrinkage basically pushes the coefficients to zero in the order of their magnitude of correlation with the response. Sometimes there are high correlation amongst the covariates (not with response), sometimes called collinearity. In this case, lasso selects only one variable $\endgroup$ – TPArrow Apr 5 at 14:46
  • $\begingroup$ This question and answer may be useful: stats.stackexchange.com/questions/154825/… $\endgroup$ – Matthew Drury Apr 5 at 14:58
  • $\begingroup$ @MatthewDrury that does look useful. I will need some time to study it. Thanks for the link. $\endgroup$ – PM. Apr 5 at 15:10
  • $\begingroup$ @TPArrow maybe you could change your comment to an answer. Do you have a reference to a text or paper that discuses this in more detail? I have Elements of Statistical Learning but it wasn't clear where this precise question was discussed. Thanks. $\endgroup$ – PM. Apr 7 at 8:29
1
$\begingroup$

Lasso is designed for several purposes, one is kind of variable selection. Roughly speaking, lasso pushes the covariates toward zero (with the use of a tuning paramere) in the order of their correlation magnitude with the response. There is a point that there is no penalisation at all (very left of your plot) and there is a point (very right in your plot) that lasso highly penalises the coefficients so that even high correlations cannot survive!

If you are not interested in the technical things, you can safely skip this paragraph. To understand lasso (in a simple linear model), you should think about the covariance matrix $(X'X/n)$. This covariance matrix sometimes (especially when p>>n [is called a high-dimensional problem] or even as simple as p>n [n=sample size, p=#covariates]) is singular. That is simply the covariance matrix is not invertible! Remember that the solution to the linear model is a function of covariance matrix, $(X'X)^{-1}X'y$. As a result, there is not a unique solution to the linear problem! What lasso does is to add a bit to this singular matrix and make it invertible. Job done! Designing the bit is not the answer to this question then I refer you to the references. The take-home message of this paragraph is that lasso basically works on the covariance matrix (or can say correlation matrix if you have already normalised your data). As a result, the decay rate in your plots is a function of the correlation. So the high correlation = late decay.

There is another scenario that lasso is useful. When there is collinearity (when covariates are intercorrelated) that lasso selects one covariate that has the most correlation with the response, provided the LARS algorithm is used. From this, this feature is a kind of algorithm dependent.

To see more details, you may want to see the original paper of Tibshirani 1996 or LARS algorithm or for a summary of lasso evolvement till 2016, see Ch1 of here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.