I watched the lecture about Lasso and at the end of this module (between 00:40 and 01:25) she explains how to choose the regularization parameter Lambda and it sounds like using (K-fold)Cross Validation technique is not a good option for Lasso. But, I don't understand why? What's the problem?


5 Answers 5


So, the point is that when you define an optimal value of $\lambda$ you must ask optimal for what? In the case of the LASSO, there are two possible goals:

  1. Estimate $\lambda_{\text{pred}}$, the value of $\lambda$ that leads to the best prediction error.

  2. Estimate $\lambda_{\text{ms}}$, the value of $\lambda$ that produces the correct model (or at least something that is close to it).

As Dr. Fox correctly notes, in general it is not the case that $\lambda_{\text{pred}} = \lambda_{\text{ms}}$, and typically $\lambda_{\text{pred}} < \lambda_{\text{ms}}$. But choosing $\lambda$ by cross-validation is using prediction error, and hence one would expect it to estimate $\lambda_{\text{pred}}$. Consequently, if you choose $\lambda$ by cross validation, you may select a $\lambda$ which leads to a model which is too big. If your goal is recovery of the true model, it follows that one should be careful applying cross validation.

I personally encounter this issue a lot when writing papers whenever I do a simulation study looking at the lasso for variable selection. Invariably, using cross-validation to select $\lambda$ is a disaster. I have had much better luck applying Lasso$(\lambda)$ to select the model and then fitting by least squares, then applying cross-validation to this entire procedure to select $\lambda$. It's still not ideal, but it is a big improvement.

That's not to say that cross-validation is completely off the table for model selection, it's just that you need to think carefully about what $\lambda$ your method is estimating. For example, lets consider ignoring the lasso and just think of a low-dimensional linear regression. In this case, leave-one-out cross validation is known to be more or less equivalent to some variant of AIC, and AIC is well-known to be inconsistent for model selection. Similarly, BIC is generally associated with leave-$V$-out cross validation where $V$ is some function of the size of the data, and it is well-known that variants of BIC are model selection consistent. Hence, there is some way of doing cross-validation that we would expect to be consistent for model selection, but leave-one-out is not.

  • $\begingroup$ (+1) Is that generally how these terms are used: "model selection" is jargon for "find the correct model"? When you say "have had better success" with this, how do you check if you have found the correct model in a real scenario? $\endgroup$ Commented Jun 26, 2018 at 18:20
  • 2
    $\begingroup$ @MatthewDrury in the context of variable selection procedures, I would say model selection generally means "find the relevant predictors." There are problems where this might make sense to do, such as some genomics problems. In real data, you cannot check that you found the correct model obviously; you have to rely on simulation experiments to understand what the procedures are doing unfortunately. There have been some attempts to estimate things like the $F$-score in real-data, but it seems extremely tough to have faith in that. $\endgroup$
    – guy
    Commented Jun 26, 2018 at 18:28
  • $\begingroup$ @AdamO It is not my personal view that the the $\lambda$ that minimizes prediction error is different from the $\lambda$ that selects the true model, it is a mathematical fact. If your goal is prediction, using $\lambda_{\text{MS}}$ will cause your coefficients to be shrunk beyond what is optimal, and if your goal is model selection then using $\lambda_{\text{pred}}$ then will select models which are too dense. And I don't know what is endorsed in the ML sphere, I am a statistician. $\endgroup$
    – guy
    Commented Jun 26, 2018 at 18:47
  • $\begingroup$ I know this is an endorsed viewpoint in the ML sphere, but I do not endorse it. I cannot distinguish $\lambda_{pred}$ from $\lambda_{MS}$ if the basis for selecting $\lambda$ is on predictive accuracy. We prefer what you are calling $\lambda_{MS}$ because it provides sparse predictions in that many of the coefficients are set to 0. Choosing $\lambda$ in the wrong fashion makes the choice overly small and leads to overfitting. $\endgroup$
    – AdamO
    Commented Jun 26, 2018 at 18:48
  • 3
    $\begingroup$ There is huge incorrect folklore on this matter. The best predictive model with KL divergence IS the best model selection. Contrary to common belief AIC messes up on small samples rather than large samples. On large samples consistency is measured strangely with coefficients that are so small and practically should be considered 0, but instead counted as a higher dimensional model. $\endgroup$ Commented Jun 26, 2018 at 18:51

The lecturer's meaning is not entirely clear. She says:

But in the case of LASSO, I just want to mention that using these types of procedures, assessing the error on a validation set or using cross validation, it's choosing lambda that provides the best predictive accuracy. But what that ends up tending to do is choosing a lambda value that's a bit smaller than might be optimal for doing model selection.

Cross-validation can be used in two ways in LASSO: to choose an optimal $\lambda$ and to assess the predictive error. To the best of my knowledge doing these things together in a single fold is not best practices. That's because you've chosen the $\lambda$ value that's optimal for a particular cross-validation fold, this naturally leads to overfitting which favors smaller values of $\lambda$.

I think better practices would be either to apply nested CV: so that for the particular training fold, CV is applied again to find an optimal $\lambda$ in that iteration; or apply CV in two batches, first a set of optimization-CV to find an optimal $\lambda$ then fix that value when fitting LASSO models in a another batch set of validation-CV steps to assess model error.

  • $\begingroup$ I've been pondering over which of the two choices that you mentioned is best. Do you have any insight? $\endgroup$
    – 24n8
    Commented Apr 25, 2021 at 15:00
  • $\begingroup$ The latter is more computationally efficient, but I don't know what the implications of that are for accuracy. $\endgroup$
    – 24n8
    Commented Apr 25, 2021 at 15:06
  • $\begingroup$ @24n8 having had a chance to rethink the problem, the other meaning may be that since k-fold cross validation is a split-sample procedure, you identify the $\lambda$ that is optimal for an e.g. $9/10 n$ sample supposing a 90/10 train/test split. One would think the result would be picking a $\lambda$ that's TOO LARGE, relative to what would be optimal in a full sample of size $n$ validated externally, in other words the added precision should reduce the need for penalization. $\endgroup$
    – AdamO
    Commented Apr 26, 2021 at 15:43

lasso regression is capable of producing “sparse” models – models that only include a subset of the predictor variables, but you loose the control on what features are being eliminated (as zero weight) VS ridge regression, when the coefficients of each predictor are shrunken towards zero but none of them can go completely to zero. Example here

The disadvantages of LASSO, however, are that we cannot use linear algebra to find the parameters. The penalty imposes a nonlinear behavior to the objective function, so we must use an iterative solver. For features that are correlated, we have no control over which feature is eliminated. Different initial guesses may lead to different feature elimination. If the features are really correlated, this will not affect the fit quality, but it will mean some models favor one feature over another. This is less of a problem in polynomial models, but often a problem in models based on physical properties that are correlated, e.g. high melting points of materials tend to be correlated with how hard they are. With LASSO, one model could favor the melting point and another could favor the hardness.

What regularization method to choose against multicollinearity & overfitting? - It depends

In cases where only a small number of predictor variables are significant, lasso regression tends to perform better because it’s able to shrink insignificant variables completely to zero and remove them from the model. However, when many predictor variables are significant in the model and their coefficients are roughly equal then ridge regression tends to perform better because it keeps all of the predictors in the model.

Still, I consider feature_selection logically by the scientist/engineer & proper design of experiment (if possible) is the best choice for getting trustfull results for statistical inference

And besides, for Lasso linear_model there are some other techniques of Cross-Validation, not K-fold, e.g. Least Angle Regression algorithm ( LassoLarsCV) & Information-criteria based model selection (according AIC/BIC)

P.S. Often I saw recommendations to estimate different scores (R2, AIC, BIC etc) & choose the model by the aggregate value, if you doubt what scores to prefer for cross-validation. Concerning IC-score:

However, such criteria need a proper estimation of the degrees of freedom of the solution, are derived for large samples (asymptotic results) and assume the correct model is candidates under investigation. They also tend to break when the problem is badly conditioned (e.g. more features than samples).


If the goal is to perform model selection and LASSO with cross validation will not work well, then the problem is not cross validation, but the problem is LASSO which is trying to perform two tasks at once

  • model/parameter selection
  • shrinking and bias/variance optimisation

These two may not be both optimal for the same regularisation parameter. Depending on the optimal bias/variance balance, the parameter selection might be done with a smaller or larger lambda than optimal*.

One can use something like relaxed LASSO to optimize both tasks seperately (see: Advantages of doing "double lasso" or performing lasso twice?). Alternatively one can also use a different prior, like elastic net regression or a horseshoe prior.

* If the optimal shrinkage is large, then the parameter selection may be too severe (ending up with less parameters than optimal). If the optimal shrinkage is small, then the parameter selection may be too low (ending up withore parameters than optimal).


Here is a very simple explanation of why there is a difference between modeling for research and modeling for prediction. I'll get into how this relates to cross-validation and Lasso by the end.

Let's say you have three possible models for a problem:

1.) A good predictive approximation with a mixture of true and accidental variables with parameters tuned for prediction and very little theory behind it (the proverbial "black box").

2.) The true model with all of and only the true variables and perfectly tuned parameters (i.e. the ground truth).

3.) A good approximation of the true model with only true variables but not all of them and parameters that are close to the true parameters (the ground truth minus some missing information)

1 and 3 are possible in practice, 2 almost always isn't. 3 is a better approximation to understanding 2, but very often 1 is actually better at predicting future values. Why? Because the "false" variables and parameters of 1 in some sense encode information that is present in 2 but not in 3 (maybe they are confounding variables in some sense).

It is very unintuitive, because if you had model 2 you would obviously do better in both understanding and prediction. But you don't have model 2. In practice you have to choose between a functional approximation and a first principles approximation. A motorcycle is a reasonable functional approximation of a car, a car that is missing a wheel is a terrible functional approximation while still being much closer in reality to a car.

Lasso regression combined with cross-validation is a great way of generating models in the first category. The problem is that there is no principled reason to think that it will get you closer to 2 or even 3. Maybe it will or maybe it won't, but you will do better being much more aggressive in your regularization than predictive accuracy would suggest if you want to get close to 3. Of course eventually all approximations approach the ground truth, a very impressive predictive accuracy is some evidence of having a true model.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.