Timeline for LASSO and ridge from the Bayesian perspective: what about the tuning parameter?
Current License: CC BY-SA 4.0
28 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jan 16, 2021 at 20:46 | history | edited | Ben | CC BY-SA 4.0 |
deleted 2 characters in body
|
| Nov 16, 2020 at 21:50 | history | edited | Ben | CC BY-SA 4.0 |
added 143 characters in body
|
| Dec 13, 2018 at 20:49 | history | bounty ended | CommunityBot | ||
| Dec 13, 2018 at 0:38 | comment | added | Ben | @amoeba: Yeah, I think you're right. As you rightly point out, that is not a pure Bayesian approach. I guess it would be a way of sneaking the data back into the prior for the hyper-parameter, so it might give you a closer analogy to the frequentist procedure, but at the expense of dropping the pure Bayesian approach. | |
| Dec 13, 2018 at 0:05 | comment | added | amoeba | Ben, what do you think about empirical Bayes approach of optimizing the hyper-parameters? See my 1st comment under the Q. I was under impression that this is a pretty standard thing to do in a Bayesian setting instead of cross-validation. It seems to directly answer @Richard's question on what is a Bayesian equivalent of CV. Of course empirical Bayes is not a fully Bayesian procedure, but I often see it used in Bayesian contexts. What's your take on it? | |
| Dec 12, 2018 at 23:32 | comment | added | Ben | ... That last formula really just establishes that model fitting with K-fold CV can be framed equivalently as a Bayesian MAP estimator, but this involves a different likelihood function in the Bayesian analysis than in the frequentist analysis. In particular, if you're doing classical model fitting via MLE with a penalty function, plus K-fold cross-validation of a tuning parameter, that is equivalent to doing Bayesian MAP with a particular prior that encompasses the penalty function (possibly improper), and a likelihood function that is adjusted to incorporate the K-fold-CV. | |
| Dec 12, 2018 at 23:29 | comment | added | Ben | @RichardHardy: It depends what you mean by use. The general idea of establishing Bayesian equivalents to other optimisation methods in classical statistics is just to check that a particular frequentist procedure also falls within the scope of Bayesian analysis, and can be given a Bayesian interpretation. Usually this doesn't involve any difference in its use. Often it will just mean that an analyst continues to use the frequentist procedure, but now does so knowing that it can be given a Bayesian interpretation as well. ... | |
| Dec 12, 2018 at 12:28 | comment | added | Richard Hardy | @Ben (ctd) My problem is that I know little about Bayes. Once it gets technical, I may easily lose the perspective. So I wonder whether this complicated analogy (the last formula) is something that is just a technical possibility or rather something that people routinely use. In other words, I am interested in whether the idea behind cross validation (here in the context of penalized estimation) is resounding in the Bayesian world, whether its advantages are utilized there. Perhaps this could be a separate question, but a short description will suffice for this particular case. | |
| Dec 12, 2018 at 12:24 | comment | added | Richard Hardy | @Ben, thanks for your explicit answer and the subsequent clarifications. You have once again done a wonderful job! Regarding 3., yes, it was quite a jump; it certainly is not a strict logical conclusion. But looking at your points w.r.t. 2. (that a Bayesian method can approximate the frequentist penalized optimization with cross validation), I no longer think that Bayesian must be "inferior". The last quibble on my side is, could you perhaps explain how the last, complicated formula could arise in practice in the Bayesian paradigm? Is it something people would normally use or not? | |
| Dec 12, 2018 at 12:19 | vote | accept | Richard Hardy | ||
| Dec 11, 2018 at 23:44 | comment | added | Ben | @RichardHardy: Thanks for these detailed comments. 1. I have now amended this part to be clearer on what I meant (i.e., that there is no Bayesian equivalent if the frequentist decides to use a hypothesis test). 2. I have now answered this in the third section. 3. I don't know how you jump from "quite effective" to "superior to Bayesian methods". If you want to assert the superiority of the frequentist methods over Bayesian methods, I think that would need to be established by detailed comparisons of properties, simulations, etc. It is possible that both methods are effective. | |
| Dec 11, 2018 at 23:38 | comment | added | Ben | @statslearner2: I have now added the specific forms of LASSO and Ridge regressions in the first section. | |
| Dec 11, 2018 at 23:38 | history | edited | Ben | CC BY-SA 4.0 |
Update
|
| Dec 11, 2018 at 23:05 | comment | added | user227843 | Thanks Ben, could you connect this more with the specific case of Lasso and Ridge? What types of priors would give us a "CV-like" result in these cases? | |
| Dec 9, 2018 at 6:29 | comment | added | hejseb | @RichardHardy My experience from selecting hyperparameters in VARs by empirical Bayes is that as long as you’re in a good neighborhood, the specific choice isn’t important for predictive accuracy. I would suspect that in the same arguably loosely-defined fashion CV helps you find a good neighborhood. But the precise value may be of less importance. | |
| Dec 9, 2018 at 2:06 | comment | added | Ben | @statslearner2: I have now added a section relating directly to the MAP analogy to k-fold cross-validation. | |
| Dec 9, 2018 at 2:05 | history | edited | Ben | CC BY-SA 4.0 |
added 4978 characters in body
|
| Dec 8, 2018 at 17:10 | comment | added | Richard Hardy | 3. Since regularization with cross validation seems to be quite effective for, say, prediction, doesn't point 2. suggest that the Bayesian approach is somehow inferior? | |
| Dec 8, 2018 at 17:09 | comment | added | Richard Hardy | 1. I do not get how (since frequentists generally use classical hypothesis tests, etc., which have no Bayesian equivalent) connects to the rest of what I or you are saying; parameter tuning has nothing to do with hypothesis tests, or does it? 2. Do I understand you correctly that there is no Bayesian equivalent to frequentist regularized estimation when the tuning parameter is selected by cross validation? What about empirical Bayes that amoeba mentions in the comments to the OP? | |
| Dec 7, 2018 at 7:04 | comment | added | user227843 | Ok +1 already, but for the bounty I'm looking for these more precise answers. | |
| Dec 7, 2018 at 7:02 | comment | added | Ben | Okay, I think I understand what you want now. Let me think about it and I'll update this answer later if I have anything useful to say on that. | |
| Dec 7, 2018 at 6:57 | history | edited | Ben | CC BY-SA 4.0 |
edited body
|
| Dec 7, 2018 at 6:57 | comment | added | user227843 | I'm looking for numerical equivalent MAP estimates. For instance, for a fixed penalty Ridge there is a gaussian prior that will give me the MAP estimate exactly equal the ridge estimate. Now, for k-fold CV ridge, what is the hyper-prior that would give me the MAP estimate which is similar to the cv-ridge estimate? | |
| Dec 7, 2018 at 6:56 | history | edited | Ben | CC BY-SA 4.0 |
Edited to clarify secnd question
|
| Dec 7, 2018 at 6:55 | comment | added | Ben | @statslearner: It won't be the same as frequentist cross-validation tests. Bayesians use their own methods, which are analogous to certain types of classical optimisation, but are not analogous to classical hypothesis testing. I have edited the question to make this clearer. | |
| Dec 7, 2018 at 6:43 | comment | added | user227843 | Also, can you make you answer specific to LASSO and Ridge, and give the specific parameterizations of the priors that would give the "k-fold-CV-like" behavior? | |
| Dec 7, 2018 at 6:41 | comment | added | user227843 | Can you explain how the hyper-prior is going to result in a similar MAP estimate as k-fold CV? This is not clear to me. | |
| Dec 7, 2018 at 6:05 | history | answered | Ben | CC BY-SA 4.0 |