Timeline for L1 Regularization vs Constraint
Current License: CC BY-SA 4.0
6 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Aug 30, 2020 at 16:48 | comment | added | jld | @user37344 if the provided value of $M$ makes the global optimum infeasible then solving the first problem is what we do to solve the second problem (since it’s the Lagrangian), so it’s not that one is harder than the other but rather that the constrained problem’s solution may not have shrinkage if the unconstrained solution just happens to be feasible and if I’m doing this, at least, I want the shrinkage | |
| Aug 30, 2020 at 16:43 | vote | accept | CommunityBot | ||
| Aug 30, 2020 at 16:43 | comment | added | user291435 | @jid Gotcha, and on top of that my understanding is that constrained optimizations tend to be numerically harder anyway, so I guess you wouldn't expect to see the 2nd form being very popular. | |
| Aug 30, 2020 at 16:37 | comment | added | jld | @user37344 In most practical cases the global minimum probably won’t be feasible so the difference won’t matter, but yeah for me personally I’d always use the first form because I want to actually get shrinkage. For any application I’ve worked on I don’t actually care about the $L_1$ norm as such, but rather I want to get the sparsity that comes from the constraint being active | |
| Aug 30, 2020 at 16:34 | comment | added | user291435 | I appreciate all of the background, that will be helpful. But regarding the main question, is the answer basically that you would use the first form to avoid an inactive constraint, whereas you might use the constrained form when returning a global minimum is acceptable/desirable? | |
| Aug 30, 2020 at 16:22 | history | answered | jld | CC BY-SA 4.0 |