What is the boundary curve for $λ_1$ and $λ_2$ that give at least a 0 component in elastic net?

Question

Define the elastic net estimate:

$ \hat{\beta}^{\lambda_1, \lambda_2} = \arg \min_{\beta \in \mathbb{R}^p} \left( \frac{1}{2n} \| y - X\beta \|_2^2 + \lambda_1 \ \frac{1}{2} \|\beta \|_2^2 + \lambda_2 \|\beta \|_1 \right) $

For lasso, when $\lambda_1 = 0$, we can compute the minimal value of $\lambda_2$ at which at least one coefficient is non-zero, and the minimal value of $\lambda_2$ at which all coefficients are non-zero, as shown in these posts: How many zeroes from lasso linear regression?, What is the smallest $\lambda$ that gives a 0 component in lasso?.

For the elastic net we can consider the boundary between sparse and non-sparse solutions as a curve of values $\lambda_1, \lambda_2$

• Can a closed form solution of $\lambda_1 \neq 0, \lambda_2$ be found at which any component of $\hat{\beta}^{\lambda_1, \lambda_2}$ is zero?
• Can a closed form solution of $\lambda_1 \neq 0, \lambda_2$ be found at which all components are zero?

Answer 1

The smallest penalty $\lambda_2$ at which all components are zero

The closed form solution of the smallest penalty $\lambda_2$ at which all components are zero, that remains the same independent from the $\lambda_1$ parameter.

This level $\lambda_{\text{all zero}}$ depends on a balance between decreasing the sum of squared residuals and the penalties, in the point $\beta = 0$. The ridge regression penalty has a zero slope at that point $\frac{\partial}{\partial \beta} \frac{1}{2} \|\beta \|_2^2 = 0$ and plays no role here. See also in: The limit of "unit-variance" ridge regression estimator when $\lambda\to\infty$

• In normal ridge regression you have a zero slope (in all directions) for $|\beta|^2$ in the point $\beta=0$. So for all finite $\lambda$ the solution can not be $\beta = 0$ (since an infinitesimal step can be made to reduce the sum of squared residuals without increasing the penalty).
• For LASSO this is not the same since: the penalty is $\lvert \beta \rvert_1$ (so it is not quadratic with zero slope). Because of that LASSO will have some limiting value $\lambda_{lim}$ above which all the solutions are zero because the penalty term (multiplied by $\lambda$) will increase more than the residual sum of squares decreases.

So for this smallest penalty level the elastic net level is the same as the lasso level.

The smallest penalty $\lambda_2$ at which at least one component is zero

...this part will be more difficult. Possibly some approximation can be made by computing the lasso solution, then consider the directional derivative/slope of the ridge penalty in that point and subtract it from the required penalty for the lasso case.