Why using cross validation is not a good option for Lasso regression? I watched the lecture about Lasso and at the end of this module (between 00:40 and 01:25) she explains how to choose the regularization parameter Lambda and it sounds like using  (K-fold)Cross Validation technique is not a good option for Lasso. But, I don't understand why? What's the problem?
 A: The lecturer's meaning is not entirely clear. She says:

But in the case of LASSO, I just want to mention that using these types of procedures, assessing the error on a validation set or using cross validation, it's choosing lambda that provides the best predictive accuracy. But what that ends up tending to do is choosing a lambda value that's a bit smaller than might be optimal for doing model selection.

Cross-validation can be used in two ways in LASSO: to choose an optimal $\lambda$ and to assess the predictive error. To the best of my knowledge doing these things together in a single fold is not best practices. That's because you've chosen the $\lambda$ value that's optimal for a particular cross-validation fold, this naturally leads to overfitting which favors smaller values of $\lambda$. 
I think better practices would be either to apply nested CV: so that for the particular training fold, CV is applied again to find an optimal $\lambda$ in that iteration; or apply CV in two batches, first a set of optimization-CV to find an optimal $\lambda$ then fix that value when fitting LASSO models in a another batch set of validation-CV steps to assess model error.
A: So, the point is that when you define an optimal value of $\lambda$ you must ask optimal for what? In the case of the LASSO, there are two possible goals:


*

*Estimate $\lambda_{\text{pred}}$, the value of $\lambda$ that leads to the best prediction error. 

*Estimate $\lambda_{\text{ms}}$, the value of $\lambda$ that produces the correct model (or at least something that is close to it). 
As Dr. Fox correctly notes, in general it is not the case that $\lambda_{\text{pred}} = \lambda_{\text{ms}}$, and typically $\lambda_{\text{pred}} < \lambda_{\text{ms}}$. But choosing $\lambda$ by cross-validation is using prediction error, and hence one would expect it to estimate $\lambda_{\text{pred}}$. Consequently, if you choose $\lambda$ by cross validation, you may select a $\lambda$ which leads to a model which is too big. If your goal is recovery of the true model, it follows that one should be careful applying cross validation. 
I personally encounter this issue a lot when writing papers whenever I do a simulation study looking at the lasso for variable selection. Invariably, using cross-validation to select $\lambda$ is a disaster. I have had much better luck applying Lasso$(\lambda)$ to select the model and then fitting by least squares, then applying cross-validation to this entire procedure to select $\lambda$. It's still not ideal, but it is a big improvement. 
That's not to say that cross-validation is completely off the table for model selection, it's just that you need to think carefully about what $\lambda$ your method is estimating. For example, lets consider ignoring the lasso and just think of a low-dimensional linear regression. In this case, leave-one-out cross validation is known to be more or less equivalent to some variant of AIC, and AIC is well-known to be inconsistent for model selection. Similarly, BIC is generally associated with leave-$V$-out cross validation where $V$ is some function of the size of the data, and it is well-known that variants of BIC are model selection consistent. Hence, there is some way of doing cross-validation that we would expect to be consistent for model selection, but leave-one-out is not. 
A: Here is a very simple explanation of why there is a difference between modeling for research and modeling for prediction. I'll get into how this relates to cross-validation and Lasso by the end.
Let's say you have three possible models for a problem:
1.) A good predictive approximation with a mixture of true and accidental variables with parameters tuned for prediction and very little theory behind it (the proverbial "black box").
2.) The true model with all of and only the true variables and perfectly tuned parameters (i.e. the ground truth).
3.) A good approximation of the true model with only true variables but not all of them and parameters that are close to the true parameters (the ground truth minus some missing information)
1 and 3 are possible in practice, 2 almost always isn't. 3 is a better approximation to understanding 2, but very often 1 is actually better at predicting future values. Why? Because the "false" variables and parameters of 1 in some sense encode information that is present in 2 but not in 3 (maybe they are confounding variables in some sense).
It is very unintuitive, because if you had model 2 you would obviously do better in both understanding and prediction. But you don't have model 2. In practice you have to choose between a functional approximation and a first principles approximation. A motorcycle is a reasonable functional approximation of a car, a car that is missing a wheel is a terrible functional approximation while still being much closer in reality to a car.
Lasso regression combined with cross-validation is a great way of generating models in the first category. The problem is that there is no principled reason to think that it will get you closer to 2 or even 3. Maybe it will or maybe it won't, but you will do better being much more aggressive in your regularization than predictive accuracy would suggest if you want to get close to 3. Of course eventually all approximations approach the ground truth, a very impressive predictive accuracy is some evidence of having a true model.
