How to compute the optimal ridge regression model

Question

I found R function ridge.cv very useful.

I would like to implement the equivalent function in MATLAB. As a starting point, I used MATLAB function b0 = ridge(y,X,k,scale), however it gives completely different results. Why does this might happen? What value should I set for variable "scale" (1 or 0 and what's their difference?)? And how could I implement it from the scratch in MATLAB?

Answer 1

For your question about the MATLAB function

According to this, the "scaled" variable is just to decide if the coefficients that you get back will be returned to the scale of your data (scaled=0) or not (scaled=1). Also, from that link, I can see that the MATLAB function doesn't actually select the "best" parameter; it just returns a vector of coefficients for each parameter. From there you have to select the optimal one, which can be done via Cross-Validation (see below).

For the "from scratch" implementation of Ridge Regression

I've already implemented this in Python/C, so I might be of some help in guidelines for implementing Ridge Regression in MATLAB.

What R does is to perform Cross-Validation (that's what "cv" stands for in the R function), and from there select the optimal parameter $\lambda$ ($k$, in MATLAB's notation, but here I'll stick to $\lambda$ for practical reasons) for Ridge Regression. If you are not familiar with what Cross-Validation does, I recommend Chapter 7 on the subject in Hastie, Tibshirani & Friedmann's book. Basically, Cross-Validation tests different values for the parameter of Ridge Regression, and selects the one that has minimum prediction error. One way to do this is to do $K$-fold Cross-Validation: divide your dataset in $K$ disjoint subsets of the data and for $k=1,..,K$, fit a model with all but the $k$ subset, and test the model on the $k$-th subset. You repeat this procedure for different parameters $\lambda$, and see which one performs better. Once you have that $\lambda$, you do a full Ridge Regression fit with that parameter and you are done. This procedure, then, should be the first thing to implement in MATLAB!

Once you've understood the procedure and have it implemented, comes the question of selecting an "intelligent" grid for the ridge parameter $\lambda$. I had the same question some days ago and the answer from cardinal can help you with this issue. If you have any questions, please check Chapter 3 on the book I gave above; it is pretty good on General Linear Models ;-).