# K-fold Cross Validation and Training/CV/Test set Techniques for choosing regularization parameter of Regression

Suppose I want to fit a lasso/ridge regression to a training set. Then, I need to choose $\lambda$, the regularization parameter. To choose $\lambda$, I can use two methods:

1. K-fold Cross Validation (from An Introduction to Statistical Learning p. 227):

1. Divide the training set into K folds (randomly).
2. Choose one fold, fit with the data in K-1 folds. Do it for all K folds.
3. Average the error for each model.
4. Choose $\lambda$ that gives the lowest error.
5. Fit with all the data in original training set with $\lambda$ chosen in (4).
2. Training/Cross Validation/Test Sets method (as taught by Andrew Ng in Coursera):

1. Divide the original training set (randomly) into 3 subsets, (new) training set, cross validation set, and test set, with proportion approx. 60%, 20%, 20%.
2. Fit with the new training set for every value of $\lambda$ you determined.
3. Measure the error of the model using cross validation set for each $\lambda$.
4. Choose the model and $\lambda$ which gives the lowest cv error.
5. Test the model with $\lambda$ chosen at (4) on test set to measure the error.

Which of these methods gives lowest bias and variance when fitting lasso/ridge regression?

• Andrew Ng's class is great, but I think his way of explaining CV is somewhat confusing. Check out this thread on Coursera for a discussion on k-fold CV vs. CV as taught by Ng. – tobip Jul 3 '14 at 15:17