Interpreting coefficients and understanding logistic lasso

Question

In lasso regression, increasing the regularisation strength/shrinkage penalty eventually forces all of the regression coefficients to zero. In this instance the regression is logistic.

The plot shows the 'trajectory' of each coefficient as the regularisation strength increases along the x axis. The explanatory variables have been standardised.

Coefficients 3 and 7 (7 is the upper green curve whose label is slightly obscured) for example 'survive' the longest, i.e. take the largest penalty values to force them to zero. Other coefficients, like 9, quickly die off to zero for smaller penalty values so don't affect the response for large enough penalty values.

(1) Can one interpret this as coefficients 3 and 7 are most important in terms of association with the response?

(2) What determines the rate of decay of the coefficients with shrinkage penalty? Why do some coefficients become zero quickly, and some remain non-zero longer?

Answer 1

Lasso is designed for several purposes, one is kind of variable selection. Roughly speaking, lasso pushes the covariates toward zero (with the use of a tuning paramere) in the order of their correlation magnitude with the response. There is a point that there is no penalisation at all (very left of your plot) and there is a point (very right in your plot) that lasso highly penalises the coefficients so that even high correlations cannot survive!

If you are not interested in the technical things, you can safely skip this paragraph. To understand lasso (in a simple linear model), you should think about the covariance matrix $(X'X/n)$. This covariance matrix sometimes (especially when p>>n [is called a high-dimensional problem] or even as simple as p>n [n=sample size, p=#covariates]) is singular. That is simply the covariance matrix is not invertible! Remember that the solution to the linear model is a function of covariance matrix, $(X'X)^{-1}X'y$. As a result, there is not a unique solution to the linear problem! What lasso does is to add a bit to this singular matrix and make it invertible. Job done! Designing the bit is not the answer to this question then I refer you to the references. The take-home message of this paragraph is that lasso basically works on the covariance matrix (or can say correlation matrix if you have already normalised your data). As a result, the decay rate in your plots is a function of the correlation. So the high correlation = late decay.

There is another scenario that lasso is useful. When there is collinearity (when covariates are intercorrelated) that lasso selects one covariate that has the most correlation with the response, provided the LARS algorithm is used. From this, this feature is a kind of algorithm dependent.

To see more details, you may want to see the original paper of Tibshirani 1996 or LARS algorithm or for a summary of lasso evolvement till 2016, see Ch1 of here