Reading through Andrew Ng's cs229 notes he shows here on page 10 how you can minimize a loss function $J\left( \theta \right)$ representing the sum of least squares by taking the gradient with respect to the weight $\theta$ to receive the expression $X^{T}X\theta - X^{T}\vec{y}$. My question is, what expression do you receive when minimizing the elastic net loss function $J\left( \theta \right) = \left\| X\theta-\vec{y}\right\|^2 + \lambda_1\left\| \theta\right\|+ \lambda_2\left\| \theta\right\|^2$ in a similar fashion?

Looking online, the best that I found is this which near the end gives you a normal equation for $\theta$ if your loss function has ridge regression.

  • 1
    $\begingroup$ Do you mean a convenient formula for $\hat\beta_{\text{elastic net}}$ like how ordinary least squares has $\hat \beta_{ols}=(X^TX)^{-1}X^Ty?$ $\endgroup$
    – Dave
    Sep 10 at 1:59
  • $\begingroup$ Yes @Dave, that is correct $\endgroup$
    – Redux
    Sep 10 at 2:00
  • $\begingroup$ I may have been incorrect about LASSO lacking a closed-form solution. I always thought it lacked one, though, so I’m not sure what I read. $\endgroup$
    – Dave
    Sep 10 at 2:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.