Maximum penalty for ridge regression

Question

Consider a regression model

$$ y = X \beta + \varepsilon. $$

I will use ridge regression to estimate $\beta$. Ridge regression contains a tuning parameter (the penalty intensity) $\lambda$. If I were given a grid of candidate $\lambda$ values, I would use cross validation to select the optimal $\lambda$. However, the grid is not given, so I need to design it first. For that I need to choose, among other things, a maximum value $\lambda_{max}$.

Question: How do I sensibly choose $\lambda_{max}$ in ridge regression?

There needs to be a balance between

• a $\lambda_{max}$ that is "too large", leading to wasteful computations when evaluating the performance of (possibly many) models that are penalized too harshly;
• a $\lambda_{max}$ that is "too small" leading to a forgone opportunity to penalize more intensely and get better performance.

(Note that the answer is simple in the case of LASSO; there you take $\lambda_{max}$ such that all coefficients are set exactly to zero for any $\lambda \geq \lambda_{max}$.)

Answer 1

The effect of $\lambda$ in the ridge regression estimator is that it "inflates" singular values $s_i$ of $X$ via terms like $(s^2_i+\lambda)/s_i$. Specifically, if SVD of the design matrix is $X=USV^\top$, then $$\hat\beta_\mathrm{ridge} = V \frac{S}{S^2+\lambda I} U^\top y.$$ This is explained multiple times on our website, see e.g. @whuber's detailed exposition here: The proof of shrinking coefficients using ridge regression through "spectral decomposition".

This suggests that selecting $\lambda$ much larger than $s_\mathrm{max}^2$ will shrink everything very strongly. I suspect that $$\lambda=\|X\|_2^2=\sum s_i^2$$ will be too big for all practical purposes.

I usually normalize my lambdas by the squared Frobenius norm of $X$ and have a cross-validation grid that goes from $0$ to $1$ (on a log scale).

---

Having said that, no value of lambda can be seen as truly "maximum", in contrast to the lasso case. Imagine that predictors are exactly orthogonal to the response, i.e. that the true $\beta=0$. Any finite value of $\lambda<\infty $ for any finite value of sample size $n$ will yield $\hat \beta \ne 0$ and hence could benefit from stronger shrinkage.

Answer 2

Maybe not quite answering your question, but instead of using ridge regression with a fixed penalization of your coefficients it would be better to use iterated adaptive ridge regression though, as the latter approximates L0 penalized regression (aka best subset), where the log likelihood of a GLM model is penalized based on a multiple of the number of nonzero coefficients in the model - see Frommlet & Noel 2016. This has the advantage that you don't have to tune the regularization level lambda at all then. Instead, you can then a priori set the regularization level $lambda$ to either $lambda = 2$ if you would like to directly optimize AIC (roughly coinciding with minimizing prediction error) or to $lambda=log(n)$ to optimize BIC (resulting in asymptotically optimal model selection in terms of selection consistency). This is what is done in the l0ara R package. To me this make more sense than first optimizing your coefficients under one objective (e.g. ridge), only to then tune the regularization level of that model based on some other criterion (e.g. minimizing cross validation prediction error, AIC or BIC). The other advantage of L0-penalized regression over ridge regression or LASSO regression is that it gives you unbiased estimates, so you can get rid of the bias-variance tradeoff that plagues most penalized regression approaches. Plus like ridge it also works for high-dimensional problems with $p>n$.

If you would like to stick with regular ridge regression, then this presentation gives a good overview of the strategies you can use to tune the ridge penalization factor. Information criteria such as AIC or BIC can also be used to tune regularisation, and they each asymptotically approximate a particular form of cross validation:

• AIC approximately minimizes the prediction error and is asymptotically equivalent to leave-1-out cross-validation (LOOCV) (Stone 1977); LOOCV in turn is approximated by generalized cross validation (GCV), but LOOCV should always be better than GCV. AIC is not consistent though, which means that even with a very large amount of data ($n$ going to infinity) and if the true model is among the candidate models, the probability of selecting the true model based on the AIC criterion would not approach 1.
• BIC is an approximation to the integrated marginal likelihood $P(D|M,A) (D=Data, M=model, A=assumptions)$, which under a flat prior is equivalent to seeking the model that maximizes $P(M|D,A)$. Its advantage is that it is consistent, which means that with a very large amount of data ($n$ going to infinity) and if the true model is among the candidate models, the probability of selecting the true model based on the BIC criterion would approach 1. This would come at a slight cost to prediction performance though if $n$ were small. BIC is also equivalent to leave-k-out cross-validation (LKOCV) where $k=n[1−1/(log(n)−1)]$, with $n=$ sample size (Shao 1997). There is many different versions of the BIC though which come down to making different approximations of the marginal likelihood or assuming different priors. E.g. instead of using a prior uniform of all possible models as in the original BIC, EBIC uses a prior uniform of models of fixed size (Chen & Chen 2008) whereas BICq uses a Bernouilli distribution specifying the prior probability for each parameter to be included.

Note that the LOOCV error can also be calculated analytically from the residuals and the diagonal of the hat matrix, without having to actually carry out any cross validation. This would always be an alternative to the AIC as an asymptotic approximation of the LOOCV error.

References

Stone M. (1977) An asymptotic equivalence of choice of model by cross-validation and Akaike’s criterion. Journal of the Royal Statistical Society Series B. 39, 44–7.

Shao J. (1997) An asymptotic theory for linear model selection. Statistica Sinica 7, 221-242.