$\def\l{|\!|}$ Given the elastic net regression

$$\min_b \frac{1}{2}\l y - Xb \l^2 + \alpha\lambda \l b\l_2^2 + (1 - \alpha) \lambda \l b\l_1$$

how can an appropriate range of $\lambda$ be chosen for cross-validation?

In the $\alpha=1$ case (ridge regression) the formula

$$\textrm{dof} = \sum_j \frac{s_j^2}{s_j^2+\lambda}$$

can be used to give an equivalent degrees of freedom for each lambda (where $s_j$ are the singular values of $X$), and degrees of freedom can be chosen in a sensible range.

In the $\alpha=0$ case (lasso) we know that

$$\lambda > \lambda_{\textrm{max}} = \max_j|\sum_t y_t X_{tj}|$$

will result in all $b_j$ being zero, and $\lambda$ can be chosen in some range $(0, \lambda_\textrm{max})$.

But how to handle the mixed case?


I think you should use a range of $0$ to

$$\lambda_\text{max}^\prime = \frac{1}{1-\alpha}\lambda_\text{max}$$

My reasoning comes from extending the lasso case, and a full derivation is below. The qualifier is it doesn't capture the $\text{dof}$ constraint contributed by the $\ell_2$ regularization. If I work out how to fix that (and decide whether it actually needs fixing), I'll come back and edit it in.

Define the objective

$$f(b) = \frac{1}{2} \|y - Xb\|^2 + \frac{1}{2} \gamma \|b\|^2 + \delta \|b\|_1$$

This is the objective you described, but with some parameters substituted to improve clarity.

Conventionally, $b=0$ can only be a solution to the optimization problem $\min f(b)$ if the gradient at $b = 0$ is zero. The term $\|b\|_1$ is non-smooth though, so the condition is actually that $0$ lies in the subgradient at $b = 0$.

The subgradient of $f$ is

$$\partial f = -X^T(y - Xb) + \gamma b + \delta \partial \|b\|_1$$

where $\partial$ denotes the subgradient with respect to $b$. At $b=0$, this becomes

$$\partial f|_{b=0} = -X^Ty + \delta[-1, 1]^d$$

where $d$ is the dimension of $b$, and a $[-1,1]^d$ is a $d$-dimensional cube. So for the optimization problem to have a solution of $b = 0$, it must be that

$$(X^Ty)_i \in \delta [-1, 1]$$

for each component $i$. This is equivalent to

$$\delta > \max_i \left|\sum_j y_j X_{ij} \right|$$

which is the defintion you gave for $\lambda_\text{max}$. If $\delta = (1-\alpha)\lambda$ is now swapped in, the formula from the top of the post falls out.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.