I fear this is an ill-posed question that has been asked a million times, but what are the basic characteristics of the penalty multiplier (usually called $\lambda$ or $\alpha$) in Ridge Regression (and perhaps the LASSO)?. In particular, I'd like to ask:
- If there are any basic constraints on how large or small $\lambda$ should be in practice. Of course the basic bounds are
epsandMAX_INTwhich are the machine error and largest integer, but this is not particularly helpful. In the case of the ridge in particular, I have heard talk of effective number of degrees of freedom, which could be a starting point.
By this I mean, that one can define the effective degrees of freedom $df(\lambda)$ by:
$$
df(\lambda) = \sum_{i=1}^p \frac{\sigma_i}{\sigma_i + \lambda}
$$
where $\sigma_i$ are the eigenvalues with multiplicity of the $p \times p $ matrix $(X^TX)$ (essentially the sample correlation matrix). Note that:
$$
df(0) = p , \quad \lim_{\lambda\to+\infty} df(\lambda) = 0
$$
Thus, one could set some threshold, e.g. $0.25$ (totally arbitrary), and solve for:
$$
df(\lambda_l) = p - 0.25, \quad df(\lambda_u) = 0.25
$$
to obtain a lower bound $\lambda_l$ and upper bound $\lambda_u$. Solving this problem seems to be routine (albeit not incredibly quick) for a run of the mill root-finder (e.g. scipy.brentq in my case.)
- How one should space these values when testing or running a cross-validation approach. For example, I often see them spaced logarithmically, e.g. 0.1,1.0, 10, 100 ... I wanted to know if there was a theoretical justification for why this should be the case, or if it was just what seems to move the needle in practice.
