Coordinate Descent for the Binary Logistic Regression

Question

I am studying Binary Logistic Regression (BLR) with the LASSO penalty and am trying to solve my objective function using the coordinate descent as discussed in the paper by https://web.stanford.edu/~hastie/Papers/glmnet.pdf, Section 3.

What I am struggiling with is how to achieved from eqauation (14) to equation (15) in the paper. I have tried to use the below questions but neither of them seems to answer my question:

Coordinate descent soft-thresholding update operator for LASSO

Coordinate descent for binomial decision in elastic net for logistic regression

My objective function is given by:

$$\mathrm{argmin} \left\{\frac{1}{n}\sum_{i=1}^n -(\bf x_i^T \bf{\beta}) y_i+ \log(1+\exp(\bf x_i^T\bf{\beta})) +{\lambda}||{\beta}||_1\right\}$$

The above is the negative log-likelihood for the BLR, can someone provide a details proof of how to use the coordinate descent to solve the above?

Answer 1

I have exactly the same problem as you did and I just figured it out. There is one sentence in the paragraph between equation (14) and equation (15) in that paper: "The Newton algorithm for maximizing the (unpenalized) log-likelihood (14) amounts to iteratively reweighted least squares." And that's the key point. (14) and (15) are not strictly equal, but maximizing (14) is equivalent to maximizing (15).

Moreover, minimizing the Lp norm for the linear regression problem by the Newton algorithm is the same as the IRLS algorithm involves solving the weighted linear least squares problem.

Two resources:

http://hua-zhou.github.io/teaching/biostatm280-2019spring/slides/21-newton/newton.html#GLM https://en.wikipedia.org/wiki/Iteratively_reweighted_least_squares#cite_note-4

Hope it helps.