2
$\begingroup$

I am trying to figure out how the intercept is calculated for logistic regression lasso using coordinate descent algorithm based on this seminal paper: Friedman, J., Hastie, T. & Tibshirani, R. Regularization Paths for Generalized Linear Models via Coordinate Descent. Journal of statistical software 33, 1-22 (2010).

In the case of linear regression, the authors clearly stated that the intercept is $\hat{\beta}_0=\bar{y}$, which is very obvious to see. Then, in the logistic regression case, they defined a "working response" $z_i$ which they simply substituted for the $y_i$ for the coordinate update in the linear regression case. However, unlike linear regression, it seems to me that $\beta_0$ also needs to be updated in each loop since this "working response" directly involves both the intercept and all coefficients. The author didn't seem to give a clear clarification on how to do this. So my questions is: how is the intercept updated for the logistic regression? Do I just simply replace $y_i$ with $z_i$ and use $\hat{\beta}_0=\bar{z}$? If so, what's the mathematical justification for it? I'm not familiar with Fortran and their core code for logistic regression was written without any comment/explanation and is just too much for me to wade through.

Thanks.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

I think your suspicion is correct - the intercept term does need to be updated for all glmnets except for elastic net.

Here's a link to the glmnet fortran code. The function that fits logistic nets begins on line 2039. If you begin there, and search downwards for a0, which is the vector containing the intercept values for each $\lambda$:

# From the source comments
a0(lmu) = intercept values for each solution

you will get to:

a0(ic,k)=a0(ic,k)-dot_product(ca(1:nk,ic,k),xm(ia(1:nk)))

k is the lambda index, ca is the vector of all coefficients:

# From the source comments
ca(nx,lmu) = compressed coefficient values for each solution

And I believe xm is the sample weighted design matrix:

xm(j)=dot_product(w,x(:,j))

This looks a lot like an update rule for the intercept term, which should be of the same form as the other coefficients.

I feel your pain in looking at the FORTRAN code. It scares the crap out of me.

Improve this answer
$\endgroup$
-1
$\begingroup$

The following is probably the answer to your question.

The Lasso penalty function does not penalize the intercept $\beta_0$. The penalty function is often presented as a sum of absolutes were the enumeration under the summation symbol starts at 1.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Yes I understand that part about intercept not being penalized in lasso. But my question is how the intercept is calculated in logistic regression lasso in each update step. It's different from linear regression where $\beta_0=\bar{y}$ is always fixed and does not need to change with each coordinate update. However, for logistic regression it's very obvious that the intercept, although not penalized, also has to be changed with each coordinate update. The manuscript does not elaborate how to update the intercept for logistic regression, and hence my questions. Thanks anyway. $\endgroup$
    – aenima
    May 12, 2015 at 18:23
  • $\begingroup$ To my knowledge this is part of optimizing the log likelihood using an iterative method like gradient descent, Newton-Raphson or such. The intercept is not treated different from the other parameters. $\endgroup$
    – spdrnl
    May 12, 2015 at 18:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.