5
$\begingroup$

Say I have a feature matrix $X$ and a target $y$. I use $k$-fold cross validation to generate $k$ out-of-sample MSE curves as a function of a penalty parameter $\lambda$

$$MSE_i(\lambda) \quad (i=1,\dots,k)$$

Given these curves, how should I choose $\lambda$? Two approaches I have seen are

  1. Choose $\lambda=\lambda^*$ to minimize the average OOS mean square error.

  2. Choose the largest $\lambda$ that is within one standard error (taken over all cross validation sets) of the $\lambda$ that minimizes the average OOS mean square error.

But it seems that 1. is too optimistic (I am likely to choose an overly complex model) and 2. is too pessimistic (there is a lot of correlation between the values of $MSE_i(\lambda)$ at different points along the curve, so 1 std deviation is too much).

Is there a happy medium, or a 'best' approach?

$\endgroup$
3
$\begingroup$

Choose $λ=λ^∗$ to minimize the average OOS mean square error.

This strategy assumes you have enough independent test cases so the error on your OOS estimate is negligible.

You are right: if the error on the OOS measurements is not negligible, this can cause a bias towards too complex models. The reason is that if you compare

  • many models of varying complexity
  • that have essentially the same performance (i.e. you cannot distinguish their performance with the given validation set-up, particularly the given total no. of test cases),
  • with a performance measurement that is subject to substantial variance,

you may "skim" the variance: the best observed performance may be caused by an (accidentally) favorable split of training and test sets rather than actually better generalization performance of the model.

See e.g.: Cawley, G. C. & Talbot, N. L. C.: On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation, Journal of Machine Learning Research, 11, 2079-2107 (2010).

The next weaker assumption is that there is some non-negligible error on the OOS estimate, but essentially the individual OOS measurements (for each surrogate model) still behave independently of each other:

Choose the largest $λ$ that is within one standard error (taken over all cross validation sets) of the $λ$ that minimizes the average OOS mean square error.

Otherwise, you need to take into account that you actually have only slightly varying models (only few training cases are exchanged between any two of the surrogate models) and only a finite number of distinct test cases. This means that the usual standard error calculation would overestimate the effective number measurements ($n$) and thus underestimate the standard error.

In consequence, in this situation you should select an even less complex model.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ The linked paper indicates that the variance of a cross-validated error estimate is a more significant concern than the bias. They show that this variability leads to sometimes selecting parameters that result in underfitting and sometimes selecting parameters that result in overfitting (most clearly shown in figure 6). If they have a finding that leads to selecting parameters resulting in overfitting more often than underfitting I must be missing it. I'm not sure it makes sense to say that this paper provides evidence of a bias towards too complex models resulting from model selection. $\endgroup$ – Jeremy Coyle Jan 29 '14 at 15:36
  • $\begingroup$ @JeremyCoyle: the variance gets larger with higher complexity, as the models get unstable (the variance on the validation estimate is partly due to the variance caused by a finite number of test cases, and partly due to model instability). You can take care of that, but it is not commonly done. Moreover, you'd want to have the least complex of all the equvialent models in the Gedankenexperiment. $\endgroup$ – cbeleites unhappy with SX Jan 29 '14 at 16:45
  • $\begingroup$ Again, the paper you cited indicates that error in the OOS performance estimates can lead both to the selection of models that are too complex and to models that are insufficiently complex. If you operate under the assumption that model selection is necessarily going to select a model that's too complex and attempt to adjust for that presumed bias, it is possible that you end up taking a selected model that's already under fitting and make it even worse. This is obviously application dependent, but my preference would be to err on the side of too much complexity rather than too little. $\endgroup$ – Jeremy Coyle Jan 29 '14 at 17:00
  • $\begingroup$ @JeremyCoyle: bias does not mean that you will necessarily err on that side. I agree that the risk of overfitting depends very much on the type of problem and data at hand - I come from a field where overfitting is extremely common, and we commonly have small sample size situations with serious trouble due to model instability - and would therefore prefer to err towardy less complex models. I also agree that selecting one complexity parameter is a comparably benign situation: trouble is more likely if more hyperparameters or a more aggressive optimization scheme is used. $\endgroup$ – cbeleites unhappy with SX Jan 29 '14 at 19:26
2
$\begingroup$

Option 1 (Choose $\lambda=\lambda^*$ to minimize the average OOS mean square error) should not be over optimistic. The point of estimating the OOS error is to select to the $\lambda$ that best fits an independent set of data from the same distribution. A $\lambda$ that results in an overly complex model will overfit the training set and therefore perform poorly on the out of sample set, and so won't be selected. See Chapter 7 of Elements of Statistical Learning for a more detailed explanation of the validity of model selection based on out of sample error, and on the benefits of cross-validation.

The philosophy behind Option 2 (Choose the largest $\lambda$ that is within one standard error (taken over all cross validation sets) of the $\lambda$ that minimizes the average OOS mean square error.) is to select a model whose performance is not substantially worse than option 1, but with a simpler (more parsimonious result). Essentially you are trading accuracy of the fit for interpretability.

In summary, Option 1 is the best choice for predictive performance of your model (and closeness to the true data generating distribution. Option 2 (and similar modifications) might lead to a model that is easier to interpret and therefore more useful in describing the true distribution in a meaningful way.

| cite | improve this answer | |
$\endgroup$
  • 3
    $\begingroup$ I believe that strategy 2 is also a step towards not choosing a more complex model if its superiority cannot be proven. $\endgroup$ – cbeleites unhappy with SX Jan 29 '14 at 9:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.