1
$\begingroup$

I am implementing these four regularization techniques for linear regression of stock data in MATLAB but i noticed elastic net is just the sum of Ridge and Lasso, and i dont full understand how exactly Non Negative Garrotte Works as a regularization technique.

How does Garrotte work and why wouldnt you just always use elastic net over lasso and ridge? (Aside from computation complexity)

$\endgroup$
3
$\begingroup$

I don't know about the Garrote, but LASSO is preferred over ridge regression when the solution is believed to have sparse features because L1 regularization promotes sparsity while L2 regularization does not, and Elastic Net is preferred over LASSO because it can deal with situations when the number of features is greater than the number of samples, and with correlated features, where LASSO behaves erratically. The additional L2 terms as a preconditioner or stabilizer by introducing strong convexity, but you'll have to read about convex optimziation to appreciate that. I think the original paper by Hastie and Zou explains all this clearly, and is worth reading.

$\endgroup$
  • $\begingroup$ I'm not sure it's the case that any one method is alwayspreferred over another. Ridge regression is useful when there is not a unique solution to the least-squares estimator, i.e. in the presence of severe multicollinearity. $\endgroup$ – Sycorax Dec 7 '14 at 14:20
  • $\begingroup$ You're right about L1 vs L2, but I would always prefer elastic net over LASSO. I'll edit my post. $\endgroup$ – Emre Dec 7 '14 at 21:10
  • $\begingroup$ I think it's probably true that many would opt for elastic nets over LASSO in general circumstances, but this implicitly assumes that you're selecting the value of the "mixing" parameter $\alpha$ such that you really do have a mix of L1 and L2 penalites, and are not simply selecting the LASSO or Ridge "by accident." $\endgroup$ – Sycorax Dec 7 '14 at 21:48
  • 1
    $\begingroup$ That's a matter of hyperparameter optimization; you don't "select" it arbitrarily yourself. $\endgroup$ – Emre Dec 7 '14 at 22:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.