# What is the boundary curve for $λ_1$ and $λ_2$ that give at least a 0 component in elastic net?

Define the elastic net estimate:

$$\hat{\beta}^{\lambda_1, \lambda_2} = \arg \min_{\beta \in \mathbb{R}^p} \left( \frac{1}{2n} \| y - X\beta \|_2^2 + \lambda_1 \ \frac{1}{2} \|\beta \|_2^2 + \lambda_2 \|\beta \|_1 \right)$$

For lasso, when $$\lambda_1 = 0$$, we can compute the minimal value of $$\lambda_2$$ at which at least one coefficient is non-zero, and the minimal value of $$\lambda_2$$ at which all coefficients are non-zero, as shown in these posts: How many zeroes from lasso linear regression?, What is the smallest $\lambda$ that gives a 0 component in lasso?.

For the elastic net we can consider the boundary between sparse and non-sparse solutions as a curve of values $$\lambda_1, \lambda_2$$

• Can a closed form solution of $$\lambda_1 \neq 0, \lambda_2$$ be found at which any component of $$\hat{\beta}^{\lambda_1, \lambda_2}$$ is zero?
• Can a closed form solution of $$\lambda_1 \neq 0, \lambda_2$$ be found at which all components are zero?
• I commented in one of the other questions/posts that in the case of elastic net there is no penalty where all components are zero, but I realize now that the level for the lasso penalty where all components are zero remains the same. Commented Jun 25 at 12:55

### The smallest penalty $$\lambda_2$$ at which all components are zero

The closed form solution of the smallest penalty $$\lambda_2$$ at which all components are zero, that remains the same independent from the $$\lambda_1$$ parameter.

This level $$\lambda_{\text{all zero}}$$ depends on a balance between decreasing the sum of squared residuals and the penalties, in the point $$\beta = 0$$. The ridge regression penalty has a zero slope at that point $$\frac{\partial}{\partial \beta} \frac{1}{2} \|\beta \|_2^2 = 0$$ and plays no role here. See also in: The limit of "unit-variance" ridge regression estimator when $\lambda\to\infty$

• In normal ridge regression you have a zero slope (in all directions) for $$|\beta|^2$$ in the point $$\beta=0$$. So for all finite $$\lambda$$ the solution can not be $$\beta = 0$$ (since an infinitesimal step can be made to reduce the sum of squared residuals without increasing the penalty).
• For LASSO this is not the same since: the penalty is $$\lvert \beta \rvert_1$$ (so it is not quadratic with zero slope). Because of that LASSO will have some limiting value $$\lambda_{lim}$$ above which all the solutions are zero because the penalty term (multiplied by $$\lambda$$) will increase more than the residual sum of squares decreases.

So for this smallest penalty level the elastic net level is the same as the lasso level.

### The smallest penalty $$\lambda_2$$ at which at least one component is zero

...this part will be more difficult. Possibly some approximation can be made by computing the lasso solution, then consider the directional derivative/slope of the ridge penalty in that point and subtract it from the required penalty for the lasso case.