I am studying Binary Logistic Regression (BLR) with the LASSO penalty and am trying to solve my objective function using the coordinate descent as discussed in the paper by https://web.stanford.edu/~hastie/Papers/glmnet.pdf, Section 3.

What I am struggiling with is how to achieved from eqauation (14) to equation (15) in the paper. I have tried to use the below questions but neither of them seems to answer my question:

Coordinate descent soft-thresholding update operator for LASSO

Coordinate descent for binomial decision in elastic net for logistic regression

My objective function is given by:

$$\mathrm{argmin} \left\{\frac{1}{n}\sum_{i=1}^n -(\bf x_i^T \bf{\beta}) y_i+ \log(1+\exp(\bf x_i^T\bf{\beta})) +{\lambda}||{\beta}||_1\right\}$$

The above is the negative log-likelihood for the BLR, can someone provide a details proof of how to use the coordinate descent to solve the above?

  • $\begingroup$ You can use the quadratic approximation of the objective function. Did you try that? $\endgroup$
    – Hello
    Commented Nov 6, 2019 at 1:48
  • $\begingroup$ Do you have any solutions yet? I have the exact same question. $\endgroup$
    – Toby
    Commented Mar 27, 2020 at 7:08

1 Answer 1


I have exactly the same problem as you did and I just figured it out. There is one sentence in the paragraph between equation (14) and equation (15) in that paper: "The Newton algorithm for maximizing the (unpenalized) log-likelihood (14) amounts to iteratively reweighted least squares." And that's the key point. (14) and (15) are not strictly equal, but maximizing (14) is equivalent to maximizing (15).

Moreover, minimizing the Lp norm for the linear regression problem by the Newton algorithm is the same as the IRLS algorithm involves solving the weighted linear least squares problem.

Two resources:

http://hua-zhou.github.io/teaching/biostatm280-2019spring/slides/21-newton/newton.html#GLM https://en.wikipedia.org/wiki/Iteratively_reweighted_least_squares#cite_note-4

Hope it helps.

  • $\begingroup$ "maximizing (14) is equivalent to maximizing (15)." No, the paper says (15) is a quadratic approximation of (14). This would mean they're not generally maximized by the same point. They describe maximizing (14) by iteratively constructing and maximizing a sequence of approximations of the form in (15). $\endgroup$
    – user20160
    Commented Dec 3, 2021 at 13:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.