I am studying Binary Logistic Regression (BLR) with the LASSO penalty and am trying to solve my objective function using the coordinate descent as discussed in the paper by https://web.stanford.edu/~hastie/Papers/glmnet.pdf, Section 3.

What I am struggiling with is how to achieved from eqauation (14) to equation (15) in the paper. I have tried to use the below questions but neither of them seems to answer my question:

Coordinate descent soft-thresholding update operator for LASSO

Coordinate descent for binomial decision in elastic net for logistic regression

My objective function is given by:

argmin {$\frac{1}{n}$ $\sum_{i=1}^n $-($\bf x_i^T$ $\bf{\beta}$) $y_i$ $+ log(1+exp($($\bf x_i^T$ $\bf{\beta}$)) $+{\lambda}||{\beta}||_1$}

The above is the negative log-likelihood for the BLR, can someone provide a details proof of how to use the coordinate descent to solve the above?

  • $\begingroup$ You can use the quadratic approximation of the objective function. Did you try that? $\endgroup$ – Hello Nov 6 '19 at 1:48
  • $\begingroup$ Do you have any solutions yet? I have the exact same question. $\endgroup$ – Toby Mar 27 at 7:08

I have exactly the same problem as you did and I just figured it out. There is one sentence in the paragraph between equation (14) and equation (15) in that paper: "The Newton algorithm for maximizing the (unpenalized) log-likelihood (14) amounts to iteratively reweighted least squares." And that's the key point. (14) and (15) are not strictly equal, but maximizing (14) is equivalent to maximizing (15).

Moreover, minimizing the Lp norm for the linear regression problem by the Newton algorithm is the same as the IRLS algorithm involves solving the weighted linear least squares problem.

Two resources:

http://hua-zhou.github.io/teaching/biostatm280-2019spring/slides/21-newton/newton.html#GLM https://en.wikipedia.org/wiki/Iteratively_reweighted_least_squares#cite_note-4

Hope it helps.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.