# When is cross validation necessary to estimate a parameter?

In 2013, @Donbeo asked whether there were any theoretical results supporting use of Cross Validation to choose the lasso penalty, and was scolded in the comments for asking "a pretty generic question about generalization error and empirical risk minimization." Well, I think was a good question.

(Picture from Zou's paper referenced by @Edgar in his answer)

I know it wouldn't work out well to try to estimate $$\lambda$$ in a frequentist maximum likelihood setting. If I had to propose why, I'd say there are problems with identifiability. But if that's true, then there must be some magical property of Cross Validation (or Empirical Risk Minimization in general) that allows one to estimate it without making any other assumptions. I would appreciate any thoughts on this.

Most of all, I'd like an explanation of what types of parameters in general is Cross-Validation more suited to than traditional inference, and some rationale as to why.

P.S. This post is an interesting read about CV as it relates to empirical bayes, but it focuses more on CV's ability to counteract model misspecification.

• $\lambda$ is't a parameter of the statistical model (that is, it does not describe the DGP, data generation process) it is a parameter of the estimation method, a hyperparameter. Maximum likelihood cannot be used for estimation of parameters that do not describe the DGP. Commented Sep 30, 2020 at 13:39
• @kjetilbhalvorsen thank you for that distinction. What about empirical bayes methods that are able to estimate the amount of regularization? Commented Sep 30, 2020 at 13:42
• Just thinking further on this, I guess it's not about whether the method is empirical bayes but whether the shrinkage is built into the model itself, like in a hierarchical modeling setting. Still curious about why trying to find lambda with RMSE, for instance, would clearly fail. Commented Sep 30, 2020 at 16:03
• I'm not sure, but think it hasto do with cross-validation simulating out-of-sample validation, so doesnot depend on an assumption that the model is the truth ... Commented Sep 30, 2020 at 16:22

We don't generally consider $$\lambda$$ as a parameter in the model you want to estimate. It doesn't have an interpretation outside of the model, in terms of your actual data. Instead, we consider $$\lambda$$ as a tuning parameter or a hyperparameter. This terminology means that $$\lambda$$ affects how you estimate $$\beta$$, but you aren't interested in $$\lambda$$ itself. Every value of $$\lambda$$ produces a unique estimate of $$\beta$$, which I'll denote with $$\hat{\beta}_\lambda$$.
So the penalized least squares equation you posted describes a huge set of possible estimates of $$\beta$$. You have to choose the "best" estimate, $$\hat{\beta}_\lambda$$ according to some criteria (best prediction, best model fit, etc.). That's when you do cross-validation to fit $$\hat{\beta}_\lambda$$ on part of your dataset and check some criterion on the remaining portion.
• DGP - "data generating process." You addressed the DGP by saying $\lambda$ "doesn't have an interpretation...in terms of your data". On minimizing RMSE, I see that you're saying the optimization would probably work, just overfit like crazy. Makes sense. Commented Oct 1, 2020 at 16:04