Suppose I want to fit a lasso/ridge regression to a training set. Then, I need to choose $\lambda$, the regularization parameter. To choose $\lambda$, I can use two methods:

  1. K-fold Cross Validation (from An Introduction to Statistical Learning p. 227):

    1. Divide the training set into K folds (randomly).
    2. Choose one fold, fit with the data in K-1 folds. Do it for all K folds.
    3. Average the error for each model.
    4. Choose $\lambda$ that gives the lowest error.
    5. Fit with all the data in original training set with $\lambda$ chosen in (4).
  2. Training/Cross Validation/Test Sets method (as taught by Andrew Ng in Coursera):

    1. Divide the original training set (randomly) into 3 subsets, (new) training set, cross validation set, and test set, with proportion approx. 60%, 20%, 20%.
    2. Fit with the new training set for every value of $\lambda$ you determined.
    3. Measure the error of the model using cross validation set for each $\lambda$.
    4. Choose the model and $\lambda$ which gives the lowest cv error.
    5. Test the model with $\lambda$ chosen at (4) on test set to measure the error.

Which of these methods gives lowest bias and variance when fitting lasso/ridge regression?

  • $\begingroup$ Andrew Ng's class is great, but I think his way of explaining CV is somewhat confusing. Check out this thread on Coursera for a discussion on k-fold CV vs. CV as taught by Ng. $\endgroup$ – tobip Jul 3 '14 at 15:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.