How to select penalty parameter after cross validation?

Question

Say I have a feature matrix $X$ and a target $y$. I use $k$-fold cross validation to generate $k$ out-of-sample MSE curves as a function of a penalty parameter $\lambda$

$$MSE_i(\lambda) \quad (i=1,\dots,k)$$

Given these curves, how should I choose $\lambda$? Two approaches I have seen are

1. Choose $\lambda=\lambda^*$ to minimize the average OOS mean square error.

2. Choose the largest $\lambda$ that is within one standard error (taken over all cross validation sets) of the $\lambda$ that minimizes the average OOS mean square error.

But it seems that 1. is too optimistic (I am likely to choose an overly complex model) and 2. is too pessimistic (there is a lot of correlation between the values of $MSE_i(\lambda)$ at different points along the curve, so 1 std deviation is too much).

Is there a happy medium, or a 'best' approach?

Answer 1

Choose $λ=λ^∗$ to minimize the average OOS mean square error.

This strategy assumes you have enough independent test cases so the error on your OOS estimate is negligible.

You are right: if the error on the OOS measurements is not negligible, this can cause a bias towards too complex models. The reason is that if you compare

• many models of varying complexity
• that have essentially the same performance (i.e. you cannot distinguish their performance with the given validation set-up, particularly the given total no. of test cases),
• with a performance measurement that is subject to substantial variance,

you may "skim" the variance: the best observed performance may be caused by an (accidentally) favorable split of training and test sets rather than actually better generalization performance of the model.

See e.g.: Cawley, G. C. & Talbot, N. L. C.: On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation, Journal of Machine Learning Research, 11, 2079-2107 (2010).

The next weaker assumption is that there is some non-negligible error on the OOS estimate, but essentially the individual OOS measurements (for each surrogate model) still behave independently of each other:

Choose the largest $λ$ that is within one standard error (taken over all cross validation sets) of the $λ$ that minimizes the average OOS mean square error.

Otherwise, you need to take into account that you actually have only slightly varying models (only few training cases are exchanged between any two of the surrogate models) and only a finite number of distinct test cases. This means that the usual standard error calculation would overestimate the effective number measurements ($n$) and thus underestimate the standard error.

In consequence, in this situation you should select an even less complex model.

Answer 2

Option 1 (Choose $\lambda=\lambda^*$ to minimize the average OOS mean square error) should not be over optimistic. The point of estimating the OOS error is to select to the $\lambda$ that best fits an independent set of data from the same distribution. A $\lambda$ that results in an overly complex model will overfit the training set and therefore perform poorly on the out of sample set, and so won't be selected. See Chapter 7 of Elements of Statistical Learning for a more detailed explanation of the validity of model selection based on out of sample error, and on the benefits of cross-validation.

The philosophy behind Option 2 (Choose the largest $\lambda$ that is within one standard error (taken over all cross validation sets) of the $\lambda$ that minimizes the average OOS mean square error.) is to select a model whose performance is not substantially worse than option 1, but with a simpler (more parsimonious result). Essentially you are trading accuracy of the fit for interpretability.

In summary, Option 1 is the best choice for predictive performance of your model (and closeness to the true data generating distribution. Option 2 (and similar modifications) might lead to a model that is easier to interpret and therefore more useful in describing the true distribution in a meaningful way.