Regularized fit from summarized data: choosing the parameter Following on from my earlier question, the solution to the normal equations for ridge regression is given by:
$$\hat{\beta}_\lambda = (X^TX+\lambda I)^{-1}X^Ty$$
Could you offer any guidance for choosing the regularization parameter $\lambda$. Additionally, since the diagonal of $X^TX$ grows with the number of observations $m$, should $\lambda$ also be a function of $m$?
 A: My answer will be based on a nice review of the problem by Anders Bjorkstorm Ridge regression and inverse problems (I would recommend to read the whole article).
Part 4 in this review is dedicated to the selection of a parameter $\lambda$ in ridge regression introducing several key approaches:


*

*ridge trace corresponds to graphical analysis of $\hat{\beta}_{i,\lambda}$ against $\lambda$. A typical plot will depict unstable (for a true ill-posted problem, you have to be sure you need this regularization in any case) behavior of different $\hat{\beta}_{i,\lambda}$ estimates for $\lambda$ close to zero, and almost constant from some point (roughly we have to detect constant behavior intersection region for all of the parameters). However decision regarding where this almost constant behavior starts is somewhat subjective. Good news for this approach is that it does not require to observe $X$ and $y$.

*$L$-curve it plots Euclidean norm of the vector of estimated parameters $|\hat{{\beta}}_\lambda|$ against the residual norm $|y - X\hat{\beta}_\lambda|$. The shape is typically close to letter $L$ so there exists a corner that determines where optimal parameter belongs to (one may choose the point in $L$ curve where the latter reaches the maximum curvature, but it is better to search for Hansen's article for more details).

*For cross-validation actually a simple "leave-one-out" approach is often chosen, seeking for $\lambda$ that maximizes (or minimizes) some forecasting accuracy criterion (you have a wide range of them, RMSE and MAPE are the two to begin with). Difficulties with 2. and 3. are that you have to observe $X$ and $y$ to implement them in practice.

