Suppose I want to fit a lasso/ridge regression to a training set. Then, I need to choose $\lambda$, the regularization parameter. To choose $\lambda$, I can use two methods:

  1. K-fold Cross Validation (from An Introduction to Statistical Learning p. 227):

    1. Divide the training set into K folds (randomly).
    2. Choose one fold, fit with the data in K-1 folds. Do it for all K folds.
    3. Average the error for each model.
    4. Choose $\lambda$ that gives the lowest error.
    5. Fit with all the data in original training set with $\lambda$ chosen in (4).
  2. Training/Cross Validation/Test Sets method (as taught by Andrew Ng in Coursera):

    1. Divide the original training set (randomly) into 3 subsets, (new) training set, cross validation set, and test set, with proportion approx. 60%, 20%, 20%.
    2. Fit with the new training set for every value of $\lambda$ you determined.
    3. Measure the error of the model using cross validation set for each $\lambda$.
    4. Choose the model and $\lambda$ which gives the lowest cv error.
    5. Test the model with $\lambda$ chosen at (4) on test set to measure the error.

Which of these methods gives lowest bias and variance when fitting lasso/ridge regression?

  • $\begingroup$ Andrew Ng's class is great, but I think his way of explaining CV is somewhat confusing. Check out this thread on Coursera for a discussion on k-fold CV vs. CV as taught by Ng. $\endgroup$ – tobip Jul 3 '14 at 15:17

The two approaches are the same from a training perspective, as both use cross-validation. If you were to use the same k and the data was significantly large, there should be no difference.

The only difference is that in approach 2 you evaluate on an unseen 20% of the data.

For the second approach we use 80% of the data for training split into 60-20. So k = 20/80 = 4


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.