# Interpreting coefficients and understanding logistic lasso

In lasso regression, increasing the regularisation strength/shrinkage penalty eventually forces all of the regression coefficients to zero. In this instance the regression is logistic.

The plot shows the 'trajectory' of each coefficient as the regularisation strength increases along the x axis. The explanatory variables have been standardised. Coefficients 3 and 7 (7 is the upper green curve whose label is slightly obscured) for example 'survive' the longest, i.e. take the largest penalty values to force them to zero. Other coefficients, like 9, quickly die off to zero for smaller penalty values so don't affect the response for large enough penalty values.

(1) Can one interpret this as coefficients 3 and 7 are most important in terms of association with the response?

(2) What determines the rate of decay of the coefficients with shrinkage penalty? Why do some coefficients become zero quickly, and some remain non-zero longer?

• 1. Yes. 2. the shrinkage basically pushes the coefficients to zero in the order of their magnitude of correlation with the response. Sometimes there are high correlation amongst the covariates (not with response), sometimes called collinearity. In this case, lasso selects only one variable – TPArrow Apr 5 '19 at 14:46
• This question and answer may be useful: stats.stackexchange.com/questions/154825/… – Matthew Drury Apr 5 '19 at 14:58
• @MatthewDrury that does look useful. I will need some time to study it. Thanks for the link. – PM. Apr 5 '19 at 15:10
• @TPArrow maybe you could change your comment to an answer. Do you have a reference to a text or paper that discuses this in more detail? I have Elements of Statistical Learning but it wasn't clear where this precise question was discussed. Thanks. – PM. Apr 7 '19 at 8:29

If you are not interested in the technical things, you can safely skip this paragraph. To understand lasso (in a simple linear model), you should think about the covariance matrix $$(X'X/n)$$. This covariance matrix sometimes (especially when p>>n [is called a high-dimensional problem] or even as simple as p>n [n=sample size, p=#covariates]) is singular. That is simply the covariance matrix is not invertible! Remember that the solution to the linear model is a function of covariance matrix, $$(X'X)^{-1}X'y$$. As a result, there is not a unique solution to the linear problem! What lasso does is to add a bit to this singular matrix and make it invertible. Job done! Designing the bit is not the answer to this question then I refer you to the references. The take-home message of this paragraph is that lasso basically works on the covariance matrix (or can say correlation matrix if you have already normalised your data). As a result, the decay rate in your plots is a function of the correlation. So the high correlation = late decay.