This presentation gives a good overview of the strategies you can use to tune the ridge penalization factor. Information criteria such as AIC or BIC can also be used to tune regularisation, and they each asymptotically approximate a particular form of cross validation:
- AIC approximately minimizes the prediction error and is asymptotically equivalent to leave-1-out cross-validation (LOOCV) (Stone 1977); LOOCV in turn is approximated by generalized cross validation (GCV), but LOOCV should always be better than GCV. AIC is not consistent though, which means that even with a very large amount of data ($n$ going to infinity) and if the true model is among the candidate models, the probability of selecting the true model based on the AIC criterion would not approach 1.
- BIC is an approximation to the integrated marginal likelihood $P(D|M,A) (D=Data, M=model, A=assumptions)$, which under a flat prior is equivalent to seeking the model that maximizes $P(M|D,A)$. Its advantage is that it is consistent, which means that with a very large amount of data ($n$ going to infinity) and if the true model is among the candidate models, the probability of selecting the true model based on the BIC criterion would approach 1. This would come at a slight cost to prediction performance though if $n$ were small. BIC is also equivalent to leave-k-out cross-validation (LKOCV) where $k=n[1−1/(log(n)−1)]$, with $n=$ sample size (Shao 1997). There is many different versions of the BIC though which come down to making different approximations of the marginal likelihood or assuming different priors. E.g. instead of using a prior uniform of all possible models as in the original BIC, EBIC uses a prior uniform of models of fixed size (Chen & Chen 2008) whereas BICq uses a Bernouilli distribution specifying the prior probability for each parameter to be included.
Note that the LOOCV error can also be calculated analytically from the residuals and the diagonal of the hat matrix, without having to actually carry out any cross validation. This would always be an alternative to the AIC as an asymptotic approximation of the LOOCV error.
Instead of using ridge regression with a fixed penalization of your coefficients it would be better to use iterated adaptive ridge regression though, as the latter approximates L0 penalized regression (aka best subset), where the log likelihood of a GLM model is penalized based on a multiple of the number of nonzero coefficients in the model - see Frommlet & Noel 2016. This has the advantage that you can then directly optimize the AIC or BIC objective directly, by choosing the appropriate regularization level $lambda$, as a $lambda = 2$ then optimizes AIC and $lambda=log(n)$ would optimize BIC. This is what is done in the l0ara R package. To me this make more sense than first optimizing your coefficients under one objective (e.g. ridge), only to then tune the regularization level of that model based on some other criterion (e.g. minimizing cross validation prediction error, AIC or BIC). The other advantage of L0-penalized regression over ridge regression or LASSO regression is that it gives you unbiased estimates, so you can get rid of the bias-variance tradeoff that plagues most penalized regression approaches. Plus like ridge it also works for high-dimensional problems with $p>n$.
Stone M. (1977) An asymptotic equivalence of choice of model by cross-validation and Akaike’s criterion. Journal of the Royal Statistical Society Series B. 39, 44–7.
Shao J. (1997) An asymptotic theory for linear model selection. Statistica Sinica 7, 221-242.