Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I'm looking to implement a coordinate descent solution to the Elastic Net (Friedman et al, 'Regularization paths for generalized linear models via coordinate descent') for logistic regression. The paper explicitly gives the coordinate update for the linear case, but I'm having difficulty understanding the solution for binomial Y.
Is there any literature/has anyone implemented coordinate descent for the logistic case? That is, outside their (unparseable to me) Fortran code?
Thanks
This is an answer to the specific questions regarding the cited paper that the OP gave in a comment. This is on the details of the algorithm described in Section 3 in the paper.
For logistic regression the working response, called $z_i$ in the paper, works as the response, that is, as $y_i$, when using the quadratic approximation. So one just replaces $y_i$ by $z_i$ in the update formula (10) and proceeds as in a weighted least squares regression problem. In particular, one forms $y_i^{(j)}$ as in the linear case.
To understand the comment after formula (10), try to derive (5) from (4) and see how the assumed standardization yields the "1" in the denominator. The comment refers to the fact that in the general weighted case the weights enter in the numerator, and the "1" in the denominator in (5) has to be replaced by a weighted sum-of-squares. This sum-of-squares is actually always present, even without weights, but if one standardizes upfront (as they do it in the paper) and the weights are $1/N$, then the sum-of-squares equals 1.
Regarding that standardization for sparse $x$-values, the scaling does not alter the sparsity structure, and can be done upfront, but the centering does and has to be done "on the fly" to preserve sparsity, see also Section 2.3. The most obvious way to me is to precompute the column means and then in (9) include the centering explicitly. Then use the bilinearity of the inner products $\langle \cdot, \cdot \rangle$ in this covariance update to rewrite the update in terms of the uncentered $x$-variables and constants $N \bar{x}_j \bar{x_i}$ and $N \bar{x}_j \bar{y}$ where e.g. $\bar{x}_j$ is the mean of $x_j$. This should be doable without really affecting the efficiency of the algorithm for the unweighted least squares problem.
Note, however, that the centering is primarily a trick for the unweighted least squares problem to get rid of the intercept parameter, and it does not affect the values of the other parameters, only the computation, whereas the scaling does affect the whole solution. For logistic regression the centering does not remove the need for an intercept, and even if you center the $x$-variables upfront by the column mean, the intercept parameter will not disappear when computing a general weighted least squares solution. Hence, it is less clear that there is any reason to center, and in Section 3 the intercept parameter is also explicitly present in the formulas. The scaling is generally recommended as a default, but this can be done without the centering and one can, as mentioned above, scale the sparse matrix without changing the sparsity structure.
As a final remark, the naive updates are recommended over the covariance updates for the weighted regression updates because the weights constantly change, and the benefit of the covariance updates disappears.
If you're looking for a solution, check out the glmnet package, which was written by the authors of that paper. It contains an implementation for logistic regression.
Even if you're looking to implement your own solution, it might be a good idea to look at their code, specifically the lognet R function and the lognet FORTRAN function.
