0
$\begingroup$

Going off of the standard set up, we have $N$ observations and $P$ predictors stored in the data matrix $\mathbf{X} = \{ x_{i,j} \}$ for $i = 1, \ldots, N$ and $j = 1, \ldots, P$. The response is given by the vector $\mathbf{y} = (y_1, \ldots, y_n)$. The coefficients are given by $\beta_0$ (the intercept) and $\boldsymbol{\beta} = (\beta_1,\ldots,\beta_P)$.

In the case of using regularization (here the lasso) for linear regression, we have to minimize $$ Q(\beta_0, \boldsymbol{\beta}) = \frac{1}{2N} \| \mathbf{y} - \beta_0 \mathbf{1}- \mathbf{X} \boldsymbol{\beta} \|_2^2 + \lambda \| \boldsymbol{\beta} \|_1. $$

However, in the literature, it is almost always the case that both $\mathbf{X}$ and $\mathbf{y}$ have been centered, so $\beta_0$ can be removed from the model. That is, $$ Q(\beta_0, \boldsymbol{\beta}) = \frac{1}{2N} \| \mathbf{y} - \mathbf{X} \boldsymbol{\beta} \|_2^2 + \lambda \| \boldsymbol{\beta} \|_1. $$

$\beta_0$ is then estimated as $$ \hat{\beta}_0 = \frac{1}{N} \sum_{i=1}^N y_i. $$

However, Tibshirani (1996) says (on page 23) that for logistic regression, "we can no longer eliminate the intercept by centering the response."

The lasso for logistic regression minimizes $$ Q(\beta_0, \boldsymbol{\beta}) = \frac{1}{N} \sum_{i=1}^N \Big[ \log \left(1 + e^{\beta_0 + \boldsymbol{x}_i^T \boldsymbol{\beta}} \right) - y_i \big(\beta_0 + \boldsymbol{x}_i^T \boldsymbol{\beta} \big)\Big] + \| \boldsymbol{\beta} \|_1, $$ the regularized negative log-likelihood.

How is the intercept estimated in this setting?

EDIT

This question is too vague, so as an attempt to elaborate, I would like to know:

  • Since we are not imposing a penalty on $\beta_0$, will $\hat{\beta}_0^{lasso}$ be the same as $\hat{\beta}_0^{log}$?
  • If so, is there some special way this $\beta_0$ is chosen like for linear regression?
  • If not, how does the $\hat{\beta}_0$ change because of regularization?
$\endgroup$
  • $\begingroup$ Your penultimate sentence provides the answer: optimize $Q(\beta_0, \beta)$. That is, both $\beta_0, \beta$ are optimized simultaneously. Can you elaborate on where you run into problems? $\endgroup$ – Sycorax Aug 3 at 23:34
  • $\begingroup$ @Sycorax I'm sorry if my question sounds a little vague, I guess I am just looking for an intuitive understanding for this. I will edit my question. $\endgroup$ – Aiden Kenny Aug 4 at 1:45
0
$\begingroup$

For your three questions:

  • Since the estimation of any parameter in the model also depends on the other variables/values of the other parameters in the model, there is no reason to expect that $\hat{\beta}_0^{lasso}$ will equal $\hat{\beta}_0^{lasso}$. This is clear from the version of the normal equations for logistic regression that you can find in this answer: When is logistic regression solved in closed form?.

  • No, the intercept must be estimated iteratively together with the other parameters.

  • I don't think much can be said in general. If the constant vector consisting only of 1's which represent the intercept in the matrix formulation of the model, is orthogonal or closely so to the other predictors, probably not much. Otherwise could change a lot, I guess. Look at examples.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.