2
$\begingroup$

In their 2004 paper on elastic net regularization, Zou and Hastie present an efficient method for finding the meta-parameters by folding the $L_2$-regularization component into the OLS problem and solving a modified OLS with only $L_1$-regularization. They then rescale the obtained coefficients to compensate for double shrinkage of the coefficients.

My question is: given an optimization problem with a general objective function (i.e. non-OLS problem) with $L_1$- and $L_2$-regularization, e.g.

$ \begin{aligned} &\underset{\vec x}{\text{maximize}} && f(\vec x) - \lambda_2\lVert \vec x\rVert_2^2 - \lambda_1\lVert \vec x\rVert_1\\ \end{aligned} $

how do I search the two-dimensional meta-parameter space most efficiently?

Options:

  • Search a 2D grid of values for $(\lambda_1,\lambda_2)$
  • Find coefficients for each $\lambda_2$ and then keeping $\lambda_2$ fixed optimize $\lambda_1$

Options 2 leads to a non-optimal solution due to double shrinkage. (What they call "naive elastic net" in the case of option 3.)

Can one still apply their suggested rescaling by $(1+\lambda_2)$ to the coefficients obtained by optimizing $\lambda_1$ for each given $\lambda_2$ to compensate for the double shrinkage, even though now the $L_2$-regularisation is no longer folded into the objective function?

What is the "best" search strategy?

As an aside: Is the prediction test error function close to the optimal point $(\lambda_1^{opt}, \lambda_2^{opt})$ likely to be smooth and approximable by a paraboloid? I.e. can I find an "almost optimal" combination by knowing a few sub-optimal points from a grid search and finding the minimum of the best-fit paraboloid?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.