4
$\begingroup$

In 2013, @Donbeo asked whether there were any theoretical results supporting use of Cross Validation to choose the lasso penalty, and was scolded in the comments for asking "a pretty generic question about generalization error and empirical risk minimization." Well, I think was a good question.

enter image description here

(Picture from Zou's paper referenced by @Edgar in his answer)

I know it wouldn't work out well to try to estimate $\lambda$ in a frequentist maximum likelihood setting. If I had to propose why, I'd say there are problems with identifiability. But if that's true, then there must be some magical property of Cross Validation (or Empirical Risk Minimization in general) that allows one to estimate it without making any other assumptions. I would appreciate any thoughts on this.

Most of all, I'd like an explanation of what types of parameters in general is Cross-Validation more suited to than traditional inference, and some rationale as to why.

P.S. This post is an interesting read about CV as it relates to empirical bayes, but it focuses more on CV's ability to counteract model misspecification.

Improve this question
$\endgroup$
4
  • 5
    $\begingroup$ $\lambda$ is't a parameter of the statistical model (that is, it does not describe the DGP, data generation process) it is a parameter of the estimation method, a hyperparameter. Maximum likelihood cannot be used for estimation of parameters that do not describe the DGP. $\endgroup$
    – kjetil b halvorsen ♦
    Sep 30, 2020 at 13:39
  • 1
    $\begingroup$ @kjetilbhalvorsen thank you for that distinction. What about empirical bayes methods that are able to estimate the amount of regularization? $\endgroup$
    – Ben Ogorek
    Sep 30, 2020 at 13:42
  • $\begingroup$ Just thinking further on this, I guess it's not about whether the method is empirical bayes but whether the shrinkage is built into the model itself, like in a hierarchical modeling setting. Still curious about why trying to find lambda with RMSE, for instance, would clearly fail. $\endgroup$
    – Ben Ogorek
    Sep 30, 2020 at 16:03
  • 1
    $\begingroup$ I'm not sure, but think it hasto do with cross-validation simulating out-of-sample validation, so doesnot depend on an assumption that the model is the truth ... $\endgroup$
    – kjetil b halvorsen ♦
    Sep 30, 2020 at 16:22

1 Answer 1

Reset to default
2
$\begingroup$

We don't generally consider $\lambda$ as a parameter in the model you want to estimate. It doesn't have an interpretation outside of the model, in terms of your actual data. Instead, we consider $\lambda$ as a tuning parameter or a hyperparameter. This terminology means that $\lambda$ affects how you estimate $\beta$, but you aren't interested in $\lambda$ itself. Every value of $\lambda$ produces a unique estimate of $\beta$, which I'll denote with $\hat{\beta}_\lambda$.

So the penalized least squares equation you posted describes a huge set of possible estimates of $\beta$. You have to choose the "best" estimate, $\hat{\beta}_\lambda$ according to some criteria (best prediction, best model fit, etc.). That's when you do cross-validation to fit $\hat{\beta}_\lambda$ on part of your dataset and check some criterion on the remaining portion.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Great answer and this does indeed answer the question. One thing that's still on my mind, and perhaps out of scope, is why minimizing the RMSE with lamda and all the DGP parameters would clearly fail but you're able to do it with cross validation. I'll leave this question open for a few days. $\endgroup$
    – Ben Ogorek
    Sep 30, 2020 at 15:58
  • $\begingroup$ I don't know what DGP is short for. I don't think minimizing RMSE or any other loss metric would fail in any way. You use cross-validation to mimic having an independent dataset for validation. This prevents overfitting. $\endgroup$
    – Eli
    Sep 30, 2020 at 19:01
  • $\begingroup$ DGP - "data generating process." You addressed the DGP by saying $\lambda$ "doesn't have an interpretation...in terms of your data". On minimizing RMSE, I see that you're saying the optimization would probably work, just overfit like crazy. Makes sense. $\endgroup$
    – Ben Ogorek
    Oct 1, 2020 at 16:04
  • $\begingroup$ Yes, that's correct. $\endgroup$
    – Eli
    Oct 1, 2020 at 16:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.