1
$\begingroup$

Say I want to fit the following logistic regression model:

$E(y)=(1+e^{-f(X)})^{-1}$

where $f(X) = \beta_0+\beta_1^TX_1 +\beta_2^TX_2 $

and I want to add L1/L2 regularization on $\beta_1$ vector but not $\beta_2$. The reason is that I am doing a causal analysis on the effect of some treatments ($X_2$) on binary $y$ and I am including a large number of control variables ($X_1$) to reduce confounding. I want my estimate on $\beta_2$ to be unbiased, but at the same time, I would be concerned about overfitting if I do not regularize. Hence, I wish to regularize only the control variables.

Is there any way to achieve this in Python?

(Edits: for anyone who had similar problems, I found this answer on StackOverflow to be helpful. TLDR: Try scaling down (e.g., multiply by 0.1) variables you would like to regularize, and then fit a LogisticRegressionCV model. This will increase the scale of coefficient for control variables $X_1$, and hence the model will regularize them much more than the treatment variables $X_2$)

Improve this question
$\endgroup$
3
  • $\begingroup$ This is a good question, but more appropriate for Stackoverflow. Try posting there (I have an answer and will post there, too) $\endgroup$
    – Cam.Davidson.Pilon
    Commented Dec 16, 2020 at 17:40
  • $\begingroup$ Just use a binary mask $m$: a vector with 1s in the position of the $\beta$s that you want to penalize, and 0s for the ones you don't want to penalize. This gives a loss function that looks like $$ \min_\beta -\sum_i \left[y_i\log(\sigma(f(x_i;\beta)) + (1-y_i)\log(1-\sigma(f(x_i;\beta)) + \lambda \beta^\top (Im) \beta \right] $$ for $I$ the identity matrix. Of course, for $m$ a vector of 1s, we have the ordinary $L^2$ penalized regression, and similarly for $L^1$ regularization. $\endgroup$
    – Sycorax ♦
    Commented Dec 16, 2020 at 18:28
  • $\begingroup$ @Sycorax I know I can implement the binary mask if I write my own optimization routine (e.g., using tensorflow/mxnet/scipy), but is there an easy to do this in logistic regression using off-the-shelf packages? $\endgroup$
    – Hao Hu
    Commented Dec 16, 2020 at 23:54

1 Answer 1

Reset to default
0
$\begingroup$

For example, in the Ridge Regulirazion you will an additional term on your loss function $\lambda \sum_{i=0}^{p} w_{i}^{2}$ where the $w_{0}$ corresponds to the intercept and the rest $w_{1},...,w_{p}$ corresdpon to your covariates.

If you want to impose regularization to particular covariates in your regression, then you have to somehow change the weight terms in $\lambda \sum_{i=0}^{p} w_{i}^{2}$

In your case, you do not want to regularize the coefficient $w_{2}$ of the covariate $X_{2}$. To achive the you have to exclude the term $w_{2}$ from the $\lambda \sum_{i=0}^{p} w_{i}^{2}$.

First check that $\sum_{i=0}^{p}w_{i}^{2}$ can be written as the inner product of the vector $\underline{w}=(w_{0},w_{1},w_{2},..,w_{p})$ with itself.

So, in order to avoid penalizing the $w_{2}$ simple use the vector $\underline{w}^{'}=(w_{0},w_{1},0,w_{3},...,w_{p})$, in order to exlucde the $w_{2}$ from the penilization.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.