# Determining Intercept for Regularized Logistic Regression

Going off of the standard set up, we have $$N$$ observations and $$P$$ predictors stored in the data matrix $$\mathbf{X} = \{ x_{i,j} \}$$ for $$i = 1, \ldots, N$$ and $$j = 1, \ldots, P$$. The response is given by the vector $$\mathbf{y} = (y_1, \ldots, y_n)$$. The coefficients are given by $$\beta_0$$ (the intercept) and $$\boldsymbol{\beta} = (\beta_1,\ldots,\beta_P)$$.

In the case of using regularization (here the lasso) for linear regression, we have to minimize $$Q(\beta_0, \boldsymbol{\beta}) = \frac{1}{2N} \| \mathbf{y} - \beta_0 \mathbf{1}- \mathbf{X} \boldsymbol{\beta} \|_2^2 + \lambda \| \boldsymbol{\beta} \|_1.$$

However, in the literature, it is almost always the case that both $$\mathbf{X}$$ and $$\mathbf{y}$$ have been centered, so $$\beta_0$$ can be removed from the model. That is, $$Q(\beta_0, \boldsymbol{\beta}) = \frac{1}{2N} \| \mathbf{y} - \mathbf{X} \boldsymbol{\beta} \|_2^2 + \lambda \| \boldsymbol{\beta} \|_1.$$

$$\beta_0$$ is then estimated as $$\hat{\beta}_0 = \frac{1}{N} \sum_{i=1}^N y_i.$$

However, Tibshirani (1996) says (on page 23) that for logistic regression, "we can no longer eliminate the intercept by centering the response."

The lasso for logistic regression minimizes $$Q(\beta_0, \boldsymbol{\beta}) = \frac{1}{N} \sum_{i=1}^N \Big[ \log \left(1 + e^{\beta_0 + \boldsymbol{x}_i^T \boldsymbol{\beta}} \right) - y_i \big(\beta_0 + \boldsymbol{x}_i^T \boldsymbol{\beta} \big)\Big] + \| \boldsymbol{\beta} \|_1,$$ the regularized negative log-likelihood.

How is the intercept estimated in this setting?

EDIT

This question is too vague, so as an attempt to elaborate, I would like to know:

• Since we are not imposing a penalty on $$\beta_0$$, will $$\hat{\beta}_0^{lasso}$$ be the same as $$\hat{\beta}_0^{log}$$?
• If so, is there some special way this $$\beta_0$$ is chosen like for linear regression?
• If not, how does the $$\hat{\beta}_0$$ change because of regularization?
• Your penultimate sentence provides the answer: optimize $Q(\beta_0, \beta)$. That is, both $\beta_0, \beta$ are optimized simultaneously. Can you elaborate on where you run into problems?
– Sycorax
Aug 3, 2019 at 23:34
• @Sycorax I'm sorry if my question sounds a little vague, I guess I am just looking for an intuitive understanding for this. I will edit my question. Aug 4, 2019 at 1:45

• Since the estimation of any parameter in the model also depends on the other variables/values of the other parameters in the model, there is no reason to expect that $$\hat{\beta}_0^{lasso}$$ will equal $$\hat{\beta}_0^{lasso}$$. This is clear from the version of the normal equations for logistic regression that you can find in this answer: When is logistic regression solved in closed form?.