K-fold Cross Validation and Training/CV/Test set Techniques for choosing regularization parameter of Regression

Question

Suppose I want to fit a lasso/ridge regression to a training set. Then, I need to choose $\lambda$, the regularization parameter. To choose $\lambda$, I can use two methods:

1. K-fold Cross Validation (from An Introduction to Statistical Learning p. 227):

   1. Divide the training set into K folds (randomly).
   2. Choose one fold, fit with the data in K-1 folds. Do it for all K folds.
   3. Average the error for each model.
   4. Choose $\lambda$ that gives the lowest error.
   5. Fit with all the data in original training set with $\lambda$ chosen in (4).

2. Training/Cross Validation/Test Sets method (as taught by Andrew Ng in Coursera):

   1. Divide the original training set (randomly) into 3 subsets, (new) training set, cross validation set, and test set, with proportion approx. 60%, 20%, 20%.
   2. Fit with the new training set for every value of $\lambda$ you determined.
   3. Measure the error of the model using cross validation set for each $\lambda$.
   4. Choose the model and $\lambda$ which gives the lowest cv error.
   5. Test the model with $\lambda$ chosen at (4) on test set to measure the error.

Which of these methods gives lowest bias and variance when fitting lasso/ridge regression?

Answer 1

The two approaches are the same from a training perspective, as both use cross-validation. If you were to use the same k and the data was significantly large, there should be no difference.

The only difference is that in approach 2 you evaluate on an unseen 20% of the data.

For the second approach we use 80% of the data for training split into 60-20. So k = 20/80 = 4

Answer 2

I would normally opt for something more like the first procedure, but for many regularised models (e.g. ridge regression), leave-one-out cross-validation performance can often be evaluated (or at least approximated) as a by-product of fitting the model. This is much more efficient as you don't need to train the model K times, just the once. This includes linear models, radial basis function neural networks, kernel learning methods, splines etc.

Now both of these methods will give biased performance estimators because the model is being trained on a subset of the available data, rather than all of it. The generalisation of most methods improve with more training data, so this will be a pessimistic bias. This source of bias is likely to be greatest for the second method, where only 60% of the data is used to train the model. For K = 3 or more, each model will be trained on more data using the first method than the second.

The other source of bias is because the hyper-parameters of the model have been tuned using the same data on which it is evaluated. This only affects method one and method two has a separate test set.

The variance of method two is likely to be higher than that of method one. This is because (i) the model is trained on less data, so the estimates of the parameters will be noisy (and vary for different partitions of the data) (ii) because the data used to select the hyper-parameters is very small (only 20%) and (iii) the data used to evaluate the generalisation error is very small (again only 20%). By re-sampling, the cross-validation in method one is making better use of the data, it is all being used to select the hyper-parameters and it is all being used for performance evaluation.

However, if you want low bias and low variance, I wouldn't use either method. I would use nested cross-validation, where the outer cross-valdaition is used to get an unbiased performance estimate (or at least to eliminate the bias due to optimising the hyper-parameters) and witing each fold of the outer-cross-validation, and inner cross-validation is used to tune the hyper-parameters. It is better to think of cross-validation as a evaluating a method of fitting a model, rather than the model itself.

See the paper I wrote with Mrs Marsupial:

G. C. Cawley and N. L. C. Talbot, Over-fitting in model selection and subsequent selection bias in performance evaluation, Journal of Machine Learning Research, vol. 11, pp. 2079-2107, July 2010 (pdf)