Timeline for What is elastic net regularization, and how does it solve the drawbacks of Ridge ($L^2$) and Lasso ($L^1$)?
Current License: CC BY-SA 4.0
26 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Feb 4, 2022 at 18:37 | comment | added | Richard Hardy | @Sycorax, wow, I had not realized the age of the post and the answers either! I guess I have learned a bit over time, too, as I did not initially catch the problem with terminology. | |
| Feb 4, 2022 at 18:34 | comment | added | Sycorax♦ | @RichardHardy This is a fair point. I do want to rewrite this answer eventually, because I feel like I've learned a lot since... oh my, 2015! | |
| Feb 4, 2022 at 18:22 | comment | added | Richard Hardy | I would add that LASSO, ridge and elastic net are estimators, not models. Therefore, expressions such as $L_1$ is a true model do not make sense. This does not invalidate your general points, though. | |
| Jun 11, 2020 at 14:32 | history | edited | CommunityBot |
Commonmark migration
|
|
| May 26, 2019 at 17:42 | history | edited | Sycorax♦ | CC BY-SA 4.0 |
deleted 23 characters in body
|
| Jun 23, 2018 at 0:13 | history | edited | Sycorax♦ | CC BY-SA 4.0 |
added 600 characters in body
|
| Oct 12, 2017 at 17:36 | comment | added | user795305 | You want the fewest tuning parameters possible, to reduce the variance of the procedure. An argument against the inclusion of an $\ell_3$ norm in the penalty might just amount to an argument that $\ell_1$ or $\ell_2$ penalization is more important than $\ell_3$ penalization. In settings where you are not interested in coefficient clustering, this is true. | |
| Apr 13, 2017 at 12:44 | history | edited | CommunityBot |
replaced http://stats.stackexchange.com/ with https://stats.stackexchange.com/
|
|
| Nov 30, 2015 at 17:13 | comment | added | Sycorax♦ | I'm not certain how refine my answer satisfactorily to account for my description here, so I will leave this comment here, with the snarky recommendation that this area of statistics be the topic of further research. | |
| Nov 30, 2015 at 17:13 | comment | added | Sycorax♦ | @Scortchi Yes, I can see now that I've backed myself into a corner. However, I'm not sure that anyone has escaped this particular corner: commentators and myself seem to agree that you should use an elastic net regression except when you know it's the wrong answer, but my point is that you don't have evidence of what model is the right answer until you can compare alternatives. Others appear able to will themselves into knowing which model is right ahead of time, but this baffles me. | |
| Nov 29, 2015 at 21:33 | comment | added | Scortchi♦ | "We can test LASSO, ridge and elastic net solutions, and make a choice of a final model" - we can, but of course that itself is a new procedure, optimizing a criterion subject to random error, which may or may not perform better than LASSo, or ridge regression, or elastic net alone. | |
| Nov 29, 2015 at 21:08 | history | edited | Scortchi♦ | CC BY-SA 3.0 |
fixed typos
|
| Nov 29, 2015 at 21:04 | comment | added | Scortchi♦ | @amoeba's question was very shrewd, & I think in answering it you seem to have changed your standards somewhat. Unless you're absolutely sure that a mix of $L_1$ & $L_2$ penalization is best then why not let the data decide how much $L_3$ penalization to apply? Your arguments still seem just a bit too strong & appear to justify adding more (hyper)parameters in almost any situation. | |
| Nov 29, 2015 at 18:55 | comment | added | Sycorax♦ | @amoeba I've added a section to answer your question. I'd be interested in hearing your thoughts. | |
| Nov 29, 2015 at 18:54 | history | edited | Sycorax♦ | CC BY-SA 3.0 |
added 2504 characters in body
|
| Nov 29, 2015 at 14:26 | comment | added | Richard Hardy | Although I praised your edit above, I think there are a couple of points which I do not buy. First, the OP is very clear and precise, and the direct answer to the question should be "No" - because of the numerous qualifications; still, you kept an unqualified statement in the first sentence of your answer. Second (and less importantly), I do not buy the "unspoken context to OP's question". Your answer better suits a question "Should I normally prefer elastic net to LASSO and ridge?" rather than the actual question. | |
| Nov 28, 2015 at 21:51 | history | edited | amoeba | CC BY-SA 3.0 |
typo in the first section title
|
| Nov 28, 2015 at 21:32 | comment | added | amoeba | +1 for in-depth discussion, but let me suggest one further argument against your point of view that elastic net is uniformly better than lasso or ridge alone. Imagine that we add another penalty to the elastic net cost function, e.g. an L3 cost, with a hyperparameter $\gamma$. I don't think there is much research on that, but I would bet you that if you do a cross-validation search on a 3d parameter grid, then you will get $\gamma \ne 0$ as the optimal value. If so, would you then argue that it is always a good idea to include L3 cost too? | |
| Nov 28, 2015 at 21:02 | comment | added | Scortchi♦ | Thanks! I'll read with care & see if I have anything to comment further. But my first comment was in response to what you wrote with no caveat - always better because it includes both as special cases - rather than to what I thought you might think. | |
| Nov 28, 2015 at 18:48 | history | edited | Sycorax♦ | CC BY-SA 3.0 |
added 3423 characters in body
|
| Nov 28, 2015 at 18:47 | comment | added | Richard Hardy | Fair points, and a welcome elaboration. | |
| Nov 28, 2015 at 18:43 | history | edited | Sycorax♦ | CC BY-SA 3.0 |
added 3423 characters in body
|
| Nov 28, 2015 at 18:07 | history | edited | Sycorax♦ | CC BY-SA 3.0 |
added 504 characters in body
|
| Nov 28, 2015 at 18:06 | comment | added | Scortchi♦ | I have to dissent: introducing $\alpha$ as another hyperparameter means it has to be set or tuned & improved performance is not guaranteed - see How bad is hyperparameter tuning outside cross-validation? | |
| Nov 28, 2015 at 18:04 | comment | added | Richard Hardy | Saying that "elastic net is always preferred over lasso & ridge regression" may be a little too strong. In small or medium samples elastic net may not select pure LASSO or pure ridge solution even if the former or the latter is actually the relevant one. Given strong prior knowledge it could make sense to choose LASSO or ridge in place of elastic net. However, in absence of prior knowledge, elastic net should be the preferred solution. | |
| Nov 28, 2015 at 17:58 | history | answered | Sycorax♦ | CC BY-SA 3.0 |