2
$\begingroup$

In their 2004 paper on elastic net regularization, Zou and Hastie present an efficient method for finding the meta-parameters by folding the $L_2$-regularization component into the OLS problem and solving a modified OLS with only $L_1$-regularization. They then rescale the obtained coefficients to compensate for double shrinkage of the coefficients.

My question is: given an optimization problem with a general objective function (i.e. non-OLS problem) with $L_1$- and $L_2$-regularization, e.g.

$ \begin{aligned} &\underset{\vec x}{\text{maximize}} && f(\vec x) - \lambda_2\lVert \vec x\rVert_2^2 - \lambda_1\lVert \vec x\rVert_1\\ \end{aligned} $

how do I search the two-dimensional meta-parameter space most efficiently?

Options:

  • Search a 2D grid of values for $(\lambda_1,\lambda_2)$
  • Find coefficients for each $\lambda_2$ and then keeping $\lambda_2$ fixed optimize $\lambda_1$

Options 2 leads to a non-optimal solution due to double shrinkage. (What they call "naive elastic net" in the case of option 3.)

Can one still apply their suggested rescaling by $(1+\lambda_2)$ to the coefficients obtained by optimizing $\lambda_1$ for each given $\lambda_2$ to compensate for the double shrinkage, even though now the $L_2$-regularisation is no longer folded into the objective function?

What is the "best" search strategy?

As an aside: Is the prediction test error function close to the optimal point $(\lambda_1^{opt}, \lambda_2^{opt})$ likely to be smooth and approximable by a paraboloid? I.e. can I find an "almost optimal" combination by knowing a few sub-optimal points from a grid search and finding the minimum of the best-fit paraboloid?

$\endgroup$

1 Answer 1

-1
$\begingroup$

I think you can either use grid-search with cross-validation, or optimized parameters search (e.g. SWARM-optimized search). See, for instance: https://stackoverflow.com/questions/65338511/sklearn-correct-procedure-for-elasticnet-hyperparameter-tuning

Or: https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-018-2401-1

Below, I show the code example for parameters tuning for regression model:

from sklearn.model_selection import GridSearchCV
parametersGrid = {
                "max_iter": [1, 5, 10, 20, 50],
                'alpha'   : [0.1,0.2,0.3,0.4,0.5,0.8,1,10,20,0.01,0.001,0.0001,0.00001],
                'l1_ratio'  : [0.1,0.2,0.3,0.4,0.5,0.8,1,0.01,0.001,0.0001,0.00001],
                'tol'       : [0.0001,0.001]
}

eNet = ElasticNet()
grid = GridSearchCV(eNet, parametersGrid, scoring='r2', cv=5)
grid.fit(X_train, y_train)

y_predictions = grid.predict(X_test)
y_train_predictions = grid.predict(X_train)
```
$\endgroup$
1
  • 2
    $\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Commented Nov 27, 2023 at 12:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.