Determining Intercept for Regularized Logistic Regression

Question

Going off of the standard set up, we have $N$ observations and $P$ predictors stored in the data matrix $\mathbf{X} = \{ x_{i,j} \}$ for $i = 1, \ldots, N$ and $j = 1, \ldots, P$. The response is given by the vector $\mathbf{y} = (y_1, \ldots, y_n)$. The coefficients are given by $\beta_0$ (the intercept) and $\boldsymbol{\beta} = (\beta_1,\ldots,\beta_P)$.

In the case of using regularization (here the lasso) for linear regression, we have to minimize $$ Q(\beta_0, \boldsymbol{\beta}) = \frac{1}{2N} \| \mathbf{y} - \beta_0 \mathbf{1}- \mathbf{X} \boldsymbol{\beta} \|_2^2 + \lambda \| \boldsymbol{\beta} \|_1. $$

However, in the literature, it is almost always the case that both $\mathbf{X}$ and $\mathbf{y}$ have been centered, so $\beta_0$ can be removed from the model. That is, $$ Q(\beta_0, \boldsymbol{\beta}) = \frac{1}{2N} \| \mathbf{y} - \mathbf{X} \boldsymbol{\beta} \|_2^2 + \lambda \| \boldsymbol{\beta} \|_1. $$

$\beta_0$ is then estimated as $$ \hat{\beta}_0 = \frac{1}{N} \sum_{i=1}^N y_i. $$

However, Tibshirani (1996) says (on page 23) that for logistic regression, "we can no longer eliminate the intercept by centering the response."

The lasso for logistic regression minimizes $$ Q(\beta_0, \boldsymbol{\beta}) = \frac{1}{N} \sum_{i=1}^N \Big[ \log \left(1 + e^{\beta_0 + \boldsymbol{x}_i^T \boldsymbol{\beta}} \right) - y_i \big(\beta_0 + \boldsymbol{x}_i^T \boldsymbol{\beta} \big)\Big] + \| \boldsymbol{\beta} \|_1, $$ the regularized negative log-likelihood.

How is the intercept estimated in this setting?

EDIT

This question is too vague, so as an attempt to elaborate, I would like to know:

• Since we are not imposing a penalty on $\beta_0$, will $\hat{\beta}_0^{lasso}$ be the same as $\hat{\beta}_0^{log}$?
• If so, is there some special way this $\beta_0$ is chosen like for linear regression?
• If not, how does the $\hat{\beta}_0$ change because of regularization?

Answer 1

For your three questions:

• Since the estimation of any parameter in the model also depends on the other variables/values of the other parameters in the model, there is no reason to expect that $\hat{\beta}_0^{lasso}$ will equal $\hat{\beta}_0^{log}$.

This is clear from the version of the normal equations for logistic regression that you can find in this answer: When is logistic regression solved in closed form?.

• No, the intercept must be estimated iteratively together with the other parameters.

• I don't think much can be said in general. If the constant vector consisting only of 1's which represent the intercept in the matrix formulation of the model, is orthogonal or closely so to the other predictors, probably not much. Otherwise could change a lot, I guess. Look at examples.