# Meta-parameter search for elastic net regularization of general objective function

In their 2004 paper on elastic net regularization, Zou and Hastie present an efficient method for finding the meta-parameters by folding the $L_2$-regularization component into the OLS problem and solving a modified OLS with only $L_1$-regularization. They then rescale the obtained coefficients to compensate for double shrinkage of the coefficients.

My question is: given an optimization problem with a general objective function (i.e. non-OLS problem) with $L_1$- and $L_2$-regularization, e.g.

\begin{aligned} &\underset{\vec x}{\text{maximize}} && f(\vec x) - \lambda_2\lVert \vec x\rVert_2^2 - \lambda_1\lVert \vec x\rVert_1\\ \end{aligned}

how do I search the two-dimensional meta-parameter space most efficiently?

Options:

• Search a 2D grid of values for $(\lambda_1,\lambda_2)$
• Find coefficients for each $\lambda_2$ and then keeping $\lambda_2$ fixed optimize $\lambda_1$

Options 2 leads to a non-optimal solution due to double shrinkage. (What they call "naive elastic net" in the case of option 3.)

Can one still apply their suggested rescaling by $(1+\lambda_2)$ to the coefficients obtained by optimizing $\lambda_1$ for each given $\lambda_2$ to compensate for the double shrinkage, even though now the $L_2$-regularisation is no longer folded into the objective function?

What is the "best" search strategy?

As an aside: Is the prediction test error function close to the optimal point $(\lambda_1^{opt}, \lambda_2^{opt})$ likely to be smooth and approximable by a paraboloid? I.e. can I find an "almost optimal" combination by knowing a few sub-optimal points from a grid search and finding the minimum of the best-fit paraboloid?

I think you can either use grid-search with cross-validation, or optimized parameters search (e.g. SWARM-optimized search). See, for instance: https://stackoverflow.com/questions/65338511/sklearn-correct-procedure-for-elasticnet-hyperparameter-tuning

Below, I show the code example for parameters tuning for regression model:

from sklearn.model_selection import GridSearchCV
parametersGrid = {
"max_iter": [1, 5, 10, 20, 50],
'alpha'   : [0.1,0.2,0.3,0.4,0.5,0.8,1,10,20,0.01,0.001,0.0001,0.00001],
'l1_ratio'  : [0.1,0.2,0.3,0.4,0.5,0.8,1,0.01,0.001,0.0001,0.00001],
'tol'       : [0.0001,0.001]
}

eNet = ElasticNet()
grid = GridSearchCV(eNet, parametersGrid, scoring='r2', cv=5)
grid.fit(X_train, y_train)

y_predictions = grid.predict(X_test)
y_train_predictions = grid.predict(X_train)
$$$$
`