Suppose I want to fit a lasso/ridge regression to a training set. Then, I need to choose $\lambda$, the regularization parameter. To choose $\lambda$, I can use two methods:
K-fold Cross Validation (from An Introduction to Statistical Learning p. 227):
- Divide the training set into K folds (randomly).
- Choose one fold, fit with the data in K-1 folds. Do it for all K folds.
- Average the error for each model.
- Choose $\lambda$ that gives the lowest error.
- Fit with all the data in original training set with $\lambda$ chosen in (4).
Training/Cross Validation/Test Sets method (as taught by Andrew Ng in Coursera):
- Divide the original training set (randomly) into 3 subsets, (new) training set, cross validation set, and test set, with proportion approx. 60%, 20%, 20%.
- Fit with the new training set for every value of $\lambda$ you determined.
- Measure the error of the model using cross validation set for each $\lambda$.
- Choose the model and $\lambda$ which gives the lowest cv error.
- Test the model with $\lambda$ chosen at (4) on test set to measure the error.
Which of these methods gives lowest bias and variance when fitting lasso/ridge regression?