Following on from my earlier question, the solution to the normal equations for ridge regression is given by:

$$\hat{\beta}_\lambda = (X^TX+\lambda I)^{-1}X^Ty$$

Could you offer any guidance for choosing the regularization parameter $\lambda$. Additionally, since the diagonal of $X^TX$ grows with the number of observations $m$, should $\lambda$ also be a function of $m$?


My answer will be based on a nice review of the problem by Anders Bjorkstorm Ridge regression and inverse problems (I would recommend to read the whole article).

Part 4 in this review is dedicated to the selection of a parameter $\lambda$ in ridge regression introducing several key approaches:

  1. ridge trace corresponds to graphical analysis of $\hat{\beta}_{i,\lambda}$ against $\lambda$. A typical plot will depict unstable (for a true ill-posted problem, you have to be sure you need this regularization in any case) behavior of different $\hat{\beta}_{i,\lambda}$ estimates for $\lambda$ close to zero, and almost constant from some point (roughly we have to detect constant behavior intersection region for all of the parameters). However decision regarding where this almost constant behavior starts is somewhat subjective. Good news for this approach is that it does not require to observe $X$ and $y$.
  2. $L$-curve it plots Euclidean norm of the vector of estimated parameters $|\hat{{\beta}}_\lambda|$ against the residual norm $|y - X\hat{\beta}_\lambda|$. The shape is typically close to letter $L$ so there exists a corner that determines where optimal parameter belongs to (one may choose the point in $L$ curve where the latter reaches the maximum curvature, but it is better to search for Hansen's article for more details).
  3. For cross-validation actually a simple "leave-one-out" approach is often chosen, seeking for $\lambda$ that maximizes (or minimizes) some forecasting accuracy criterion (you have a wide range of them, RMSE and MAPE are the two to begin with). Difficulties with 2. and 3. are that you have to observe $X$ and $y$ to implement them in practice.
| cite | improve this answer | |
  • 3
    $\begingroup$ In my experience leave one out cross validation almost always results in too little regularization. $k$-fold cross validation nearly always works better. $\endgroup$ – cardinal Apr 12 '11 at 14:08
  • $\begingroup$ (+1) @cardinal, nice addition, to be frank, I have little experience with cross validation methods. The usual simple things I used in practice are jack-knives (dropping up to $k$ subsequent observations) and out-of-samples for time series data. Though $k$-fold could be implemented for some time series models too, have to try it to build my own experience first. $\endgroup$ – Dmitrij Celov Apr 12 '11 at 16:23
  • $\begingroup$ There are some nice block bootstrap methods for stationary time series. Perhaps they could or have been modified for the purposes of selecting a regularization parameter. $\endgroup$ – cardinal Apr 12 '11 at 16:32
  • $\begingroup$ You may find the following paper useful: Golub, G. H.; Heath, M. & Wahba, G. Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter. Technometrics, 1979, 21, 215-223. The criterion introduced by Golub et al. does not require any re-sampling. $\endgroup$ – emakalic Apr 13 '11 at 0:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.