3
$\begingroup$

This is (I think) a routine question, but I haven't seen an answer here. In the past I had to fit logistic regression models with linear constraints on the coefficients, in the case when some of the predictors were categorical. Until now I've been using glmnet, which by the way only allows bound constraints ($a_i\leq\beta_i\leq b_i $). However, apart from the limitation on the type of constraints, I find it annoying to be forced to use glmnet matrix/column specification for data, instead than a more expressive formula as done in glm. Also, most of the times I only needed to fit an unregularized logistic regression model ($\lambda=0$). However, glmnet help warns against fitting the model for a single value of $\lambda$. Thus I had to fit a sequence of models, parameterized by a sequence of $\lambda$, and then extract the model corresponding to $\lambda=0$. It gets boring the $n-$th time you have to do it.

For all these reasons I'm looking into other solutions to fit unregularized logistic regression models, with linear constraints on the coefficients. One could of course derive the objective function for logistic regression (as done here) and then use an optimizer in R which supports convex optimization problems and linear constraints (any suggestions?). However, this is complicated and I'm not sure how I would setup the optimization to take into account the categorical predictors. Is there a simpler way to achieve my goal? If not, can you show me in detail how to do this with optimization ?

Improve this question
$\endgroup$
4
  • $\begingroup$ Does "linear constraints" means that one of the coefficients can be expressed as a linear combination of the others? Would the approach outlined by Glen_b here (combining the predictors to satisfy the constrain) help? $\endgroup$
    – juod
    Commented Apr 18, 2017 at 19:49
  • 1
    $\begingroup$ Could you just fit a Bayesian model & specify the prior to have 0 density in a given region? $\endgroup$
    – gung - Reinstate Monica
    Commented Apr 18, 2017 at 20:08
  • 1
    $\begingroup$ @juice not necessarily. That would indeed be the case for linear equality constraints: in that case, some coefficients may be written as linear combinationsystem of the others. But I was thinking if linear inequality constraints. The coefficient vector $\mathbf{\beta}= (\beta_1,\dots,\beta_p)$ must satisfy $\mathbf{A}\mathbf{\beta}-\mathbf{b}\leq 0$, where $\mathbf{A}$ is $d\times p$, with $d$ the number of such constraints, and $p$ the number of coefficients in the model. $\endgroup$
    – DeltaIV
    Commented Apr 18, 2017 at 21:02
  • $\begingroup$ @gung good idea, I had also thought of Bayesian logistic regression, but: 1. the only R package I know of which allows non-normal priors for logistic regression is brms, and frankly installing Stan to solve this problem sounds like using a cannon to kill a fly. 2. Bayes gives me a posterior distribution. I only need point estimates for the coefficients (the model goal is prediction, not inference). Of course I can summarise $p (\theta|y)$ by mean, mode, etc. but again, I was thinking of a simple solution. Anyway, I'd accept a Bayesian answer, if there is no simpler way. $\endgroup$
    – DeltaIV
    Commented Apr 19, 2017 at 5:35

1 Answer 1

Reset to default
1
$\begingroup$

The question is not clear to me. But if you want to use formula instead of matrix input. You can use caret pacakge, which provide universal interface to many packages including glmnet. Or you can use model.matrix to convert formula to matrix form.

If you want to fit a logistic regression with $\lambda=0$, you can directly use traditional glm. If you want to manually implement it, my answer here has the code to to calculate the objective function and gradient of logistic loss and use BFGS to optimize.

For the revised question: projected gradient descent may work well, since the "box constraints" is easy to solve. My answer here gives some details on projected gradient descent. (I believe projected gradient descent is used by glmnet).

Solving constrained optimization problem: projected gradient vs. dual?

Additional reference for projected gradient descent, book from the author of glmnet: Statistical Learning with Sparsity page 120

Improve this answer
$\endgroup$
6
  • $\begingroup$ you're right, it's late and my brain is not working properly :) so the question was unclear. Please have a look at my edits and let me know if it's more comprehensible now. $\endgroup$
    – DeltaIV
    Commented Apr 18, 2017 at 18:58
  • 2
    $\begingroup$ @DeltaIV the revision looks much better! I would further suggest to remove the details on glmnet, just say, how to do logistic regression with constrain on coefficient. In addition, if you study model.matrix and this link it will answer your question about how to deal with discrete variables. (which is a separate question may cause further confusion to the original question) $\endgroup$
    – Haitao Du
    Commented Apr 18, 2017 at 19:28
  • $\begingroup$ nls is not applicable here. The objective function of logistic regression is not a sum of squares, thus constrained nonlinear least squares isn't helpful (constrained linear least squares, even less). $\endgroup$
    – DeltaIV
    Commented Apr 19, 2017 at 7:27
  • $\begingroup$ On the contrary , projected gradient descent sounds nice...your link states that a closed form solution is available for linear constraints. Can you show the closed form solution? Note that I'm talking about inequality constraints. If linear constraints are too complicated, even the closed form solution for box constraints would be ok. $\endgroup$
    – DeltaIV
    Commented Apr 19, 2017 at 7:34
  • $\begingroup$ @DeltaIV sorry my mistake, I thought it is nlm not nls. I revised my answer. $\endgroup$
    – Haitao Du
    Commented Apr 19, 2017 at 13:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.