2
$\begingroup$

In linear regression, it is well known that ridge regression shrinks the vector of coefficients towards zero as $\lambda \rightarrow \infty$ and that the lasso sets some to zero while including others in a piecewise linear fashion. I have also seen both techniques be applied to GLMs (by adding a penalty on the negative log-likelihood instead)

My question: Do the same properties hold for GLMs? Where can I find a formal proof of this? And if not, are there certain families of distributions an link functions that enjoy these properties?

$\endgroup$
1
  • 2
    $\begingroup$ Regularisation tends to shrink coefficients towards $0$ by penalising their magnitude in some way, and the greater the penalty is, the more this tends to happen. This is a general effect. But even with ordinary use you may see what look like exceptions: for example with lasso you can have a coefficient which disappears for some value of $\lambda$, reappears for a larger $\lambda$ and then disappears for an even larger value. stats.stackexchange.com/questions/154825/… illustrates an example $\endgroup$
    – Henry
    May 23, 2021 at 9:59

1 Answer 1

1
$\begingroup$

The description of ridge regression and LASSO in Chapter 6 of An Introduction to Statistical Learning (ISL) is in the context of linear regression with a residual sum of square (RSS) cost function. For ridge you minimize:

$$ \text{RSS} + \lambda \sum_{j=1}^p \beta_j^2$$

and for LASSO you minimize

$$ \text{RSS} + \lambda \sum_{j=1}^p \left| \beta_j \right|,$$

where the $\beta_j$ are the $p$ regression coefficients. These are equivalent to minimizing RSS subject to constraints $\sum_{j=1}^p \beta_j^2 \le s$ for ridge and $\sum_{j=1}^p \left| \beta_j \right| \le s$ for LASSO (equations 6.8 and 6.9 of ISL). Figure 6.7 of ISL shows how it's the different shapes of those constraints that lead ridge to penalize all coefficients and LASSO to do variable selection along with penalization. There is nothing magical about RSS as the cost function; the above properties hold for any cost function well-enough behaved to allow you to find a minimum subject to the constraint.

This page provides links for more details about convergence of GLMs. Link functions of the exponential family can be well behaved in that they have concave negative log-likelihoods generally providing a unique minimum. Extreme-enough data, however, can force an unpenalized solution to an extreme that isn't well behaved. For example, in unpenalized logistic regression models you can get perfect separation that leads to some effectively infinite coefficient estimates.

Ridge or LASSO penalizations can actually help with this situation. By restraining coefficient values (or equivalently, log-likelihoods of data given the coefficient values) away from extremes, ridge or LASSO penalization can move solutions of such models to regions of coefficient space in which the penalized cost function is well behaved.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.