# Regularized fit from summarized data: choosing the parameter

Following on from my earlier question, the solution to the normal equations for ridge regression is given by:

$$\hat{\beta}_\lambda = (X^TX+\lambda I)^{-1}X^Ty$$

Could you offer any guidance for choosing the regularization parameter $\lambda$. Additionally, since the diagonal of $X^TX$ grows with the number of observations $m$, should $\lambda$ also be a function of $m$?

Part 4 in this review is dedicated to the selection of a parameter $\lambda$ in ridge regression introducing several key approaches:
1. ridge trace corresponds to graphical analysis of $\hat{\beta}_{i,\lambda}$ against $\lambda$. A typical plot will depict unstable (for a true ill-posted problem, you have to be sure you need this regularization in any case) behavior of different $\hat{\beta}_{i,\lambda}$ estimates for $\lambda$ close to zero, and almost constant from some point (roughly we have to detect constant behavior intersection region for all of the parameters). However decision regarding where this almost constant behavior starts is somewhat subjective. Good news for this approach is that it does not require to observe $X$ and $y$.
2. $L$-curve it plots Euclidean norm of the vector of estimated parameters $|\hat{{\beta}}_\lambda|$ against the residual norm $|y - X\hat{\beta}_\lambda|$. The shape is typically close to letter $L$ so there exists a corner that determines where optimal parameter belongs to (one may choose the point in $L$ curve where the latter reaches the maximum curvature, but it is better to search for Hansen's article for more details).
3. For cross-validation actually a simple "leave-one-out" approach is often chosen, seeking for $\lambda$ that maximizes (or minimizes) some forecasting accuracy criterion (you have a wide range of them, RMSE and MAPE are the two to begin with). Difficulties with 2. and 3. are that you have to observe $X$ and $y$ to implement them in practice.
• In my experience leave one out cross validation almost always results in too little regularization. $k$-fold cross validation nearly always works better. – cardinal Apr 12 '11 at 14:08
• (+1) @cardinal, nice addition, to be frank, I have little experience with cross validation methods. The usual simple things I used in practice are jack-knives (dropping up to $k$ subsequent observations) and out-of-samples for time series data. Though $k$-fold could be implemented for some time series models too, have to try it to build my own experience first. – Dmitrij Celov Apr 12 '11 at 16:23