k-fold cross validation for LASSO regression model

Assume we have a simple linear regression model expressed as $Y= X \beta + e$.

We know that finding the regression coefficients $\beta$ using the LASSO method is performed by penalizing the Least Squares by the L1 norm penalty:

$~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~$$argmin_\beta~(Y - X\beta)^2 + \lambda ||\beta||_1$

The most known technique to find the parameter $\lambda$ is k-fold cross validation. So can anyone help me how can I apply in matlab the k-fold cross validation in order to find the values of $\lambda$?

Any help will be very appreciated!

• $[B ~~FitInfo] = lasso(X,Y,'CV',k);$ $%Plot the cross-validated fits.$ $lassoPlot(B,FitInfo,'PlotType','CV');$ – Christina Sep 2 '15 at 13:34

2 Answers

If you have N samples, then your Y matrix is of size N x 1 and your X matrix is of size N x D.

To optimize the regularization parameter $\lambda$, here is a way to do k-fold cross-validation in MATLAB :

groups = crossvalind('Kfold', N, K);


groups is a 1 x N vector, and it has values between 1 and K. So for each fold, you get training and test samples with:

for k=1:K
trains = find(groups~=k);
tests = find(groups==k);
end


Since your case is regression, you train your system (for each fold) like this :

beta = inv(X(trains,:)) * Y

and you get your test set estimates like this :

Y_test = X(tests,:) * beta


For optimization you need a measure of accuracy, which might be RMSE for regression. So you need to try different values of lambda, by looking at the error (average of K folds), you can choose the lambda that results in the minimum average error.

Hope it helps !

You can simply use lasso function in MATLAB with 'CV'=k. In the output FitInfo you can find everything you want. For details, please read the MATLAB documentation.

• We are looking for long answers. Please extend your answer. – Ferdi Jan 20 '17 at 17:38