5
$\begingroup$

I want to use an Adaptive Lasso instead of a standard Lasso because of the Oracle properties of the former. However, I cannot seem to find an option to implement an Adaptive Lasso for a logistic regression in Python. For example, the package asgl lets one implement penalties such as an Adaptive Lasso, however, it only supports linear and quantile regressions. Unfortunately, switching to R (glmnet e.g.) is no option here because of company regulation.

Does anyone know how to implement Adaptive Lasso penalties for logistic regressions in Python?

$\endgroup$

1 Answer 1

7
$\begingroup$

Adaptive LASSO is a two-step estimator; check out section 3.1 of Zou "The Adaptive Lasso and Its Oracle Properties" (2006). (This is the original paper that proposed adaptive LASSO.) You can implement the steps separately. Let $p$ be the number of regressors in your model.

  1. You start with a $\sqrt{n}$-consistent estimator of $\beta=(\beta_1,\dots,\beta_p)^\top$ such as the MLE.*
  2. For $j=1,\dots,p$, you specify $\tilde X_j$ as $\frac{X_j}{|\hat\beta_j|^\gamma}$ for some $\gamma>0$ (e.g. $\gamma=1$). You then run a standard LASSO using these modified $\tilde X$s instead of the original ones. (See Section 3.5.)

*This requires the number of regressors $p$ to be less than the sample size $n$, $p<n$. Otherwise, you need to look for another $\sqrt{n}$-consistent estimator.

$\endgroup$
1
  • 4
    $\begingroup$ (+1) So just to make things extra straightforward, in python this approach could be implemented by using LogisticRegression with the inverse regularization strength C=10000 or some other big number. Then, take each column of your data and divide it by $|\hat{\beta}_j|^\gamma$ where $\beta_j$ comes from the _coef field of the LogisticRegression object and $\gamma$ is a positive hyperparameter; perhaps $\gamma=1$. Finally, create a second LogisticRegression object using the transformed X variables and a more reasonable C (chosen in the usual manner(s)). $\endgroup$ Nov 10, 2022 at 15:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.