I fear this is an ill-posed question that has been asked a million times, but what are the basic characteristics of the penalty multiplier (usually called $\lambda$ or $\alpha$) in Ridge Regression (and perhaps the LASSO)?. In particular, I'd like to ask:

  1. If there are any basic constraints on how large or small $\lambda$ should be in practice. Of course the basic bounds are eps and MAX_INT which are the machine error and largest integer, but this is not particularly helpful. In the case of the ridge in particular, I have heard talk of effective number of degrees of freedom, which could be a starting point.

By this I mean, that one can define the effective degrees of freedom $df(\lambda)$ by: $$ df(\lambda) = \sum_{i=1}^p \frac{\sigma_i}{\sigma_i + \lambda} $$ where $\sigma_i$ are the eigenvalues with multiplicity of the $p \times p $ matrix $(X^TX)$ (essentially the sample correlation matrix). Note that: $$ df(0) = p , \quad \lim_{\lambda\to+\infty} df(\lambda) = 0 $$ Thus, one could set some threshold, e.g. $0.25$ (totally arbitrary), and solve for: $$ df(\lambda_l) = p - 0.25, \quad df(\lambda_u) = 0.25 $$ to obtain a lower bound $\lambda_l$ and upper bound $\lambda_u$. Solving this problem seems to be routine (albeit not incredibly quick) for a run of the mill root-finder (e.g. scipy.brentq in my case.)

  1. How one should space these values when testing or running a cross-validation approach. For example, I often see them spaced logarithmically, e.g. 0.1,1.0, 10, 100 ... I wanted to know if there was a theoretical justification for why this should be the case, or if it was just what seems to move the needle in practice.
  • $\begingroup$ $\lambda \ge 0$ of course. And since you know how $\lambda$ affects the singular values of the (regularized) design matrix, you can use those singular values to determine a reasonable upper bound. You can also use the alternative characterization as adjoining artificial observations to decide when $\lambda$ is too large for your purposes. It's unclear what "in practice" might mean, though. $\endgroup$
    – whuber
    Commented May 10, 2022 at 19:22
  • $\begingroup$ Thanks @whuber - just read on the effective degrees of freedom approach and it makes a lot of sense and provides a reasonable way to bound the values. $\endgroup$ Commented May 10, 2022 at 20:52
  • $\begingroup$ In practice, one begins with a reasonable range of $\lambda,$ evaluates the solutions along a grid, and studies the ridge trace plot. If the traces move too rapidly, you need a finer grid, but otherwise almost any kind of interpolation among the values will be fine. If the traces don't seem to extend far enough, you can run the algorithm out to larger values of $\lambda.$ Thus, perhaps the most useful information concerns how the estimates change with $\lambda:$ your formulas indicate they vary hyperbolically. That's enough to determine a useful spacing in the grid. $\endgroup$
    – whuber
    Commented May 10, 2022 at 20:57
  • 1
    $\begingroup$ Related: "Maximum penalty for ridge regression". $\endgroup$ Commented May 11, 2022 at 5:40


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.