5
$\begingroup$

My feature data is defined in such a way that I believe all weights must be non-negative. I am looking for a reference discussing how to optimize the weights of a logistic regression classifier with the constraint that the weights must be non-negative, and perhaps also a constraint on the sign of the bias.

Will replacing weights with squared weights and using the regular ML cost function with a local optimization scheme work?

$\endgroup$
  • $\begingroup$ did you find any theoretical advantages of setting this constraint? I also have a problem with SVM, where I need to set the weights (the W) to be non-negative. If you know any references or advantages could you please reply. $\endgroup$ – user570593 Aug 1 '15 at 15:12
4
$\begingroup$

If you are familiar with convex optimization, I suspect this can be formalized as a quadratic programming problem (or some other convex problem) and then solved with a QP solver. If this direction interests you, I can elaborate further.

Regardless of the method which is used, if the problem is convex (there are a number of ways to check this and I know that unconstrained logistic regression is convex) you are guaranteed that a local minimum will also be a global minimum.

I might be mistaken, but it seems to be convex because if the original solution space is convex and the set of positive solutions is also convex (it is a cone), the new problem would be convex since an intersection of convex sets is convex. But this is hand-waving, best to examine it formally.

$\endgroup$
  • 2
    $\begingroup$ if the problem is convex... this is exactly my question $\endgroup$ – eyaler Oct 4 '12 at 8:56
  • $\begingroup$ @eyaler Can you specify then what exactly is the logistic function you want to fit? There are different versions of logistic functions. $\endgroup$ – Bitwise Oct 4 '12 at 14:56
3
$\begingroup$

I have done this successfully using projected gradient descent.

The algorithm is very simple - take a gradient step, then set all negative coefficients to zero (i.e. project onto the feasible set).

I started with Leon Bottou's code here: http://leon.bottou.org/projects/sgd.

$\endgroup$
1
$\begingroup$

I have answered to that question in this blog post. It is about adding some dummy data but the real solution is of course to do Projected Gradient Descent in the primal as user1149913 said.

$\endgroup$

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

1
$\begingroup$

Your objective can be achieved using an R package "glmnet". It is a machine learning package in R that fits generalized linear models via penalized maximum likelihood. But, it also allows coefficients to be constrained. This can be used to solve your problem. For more details please follow the link below: https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html

Coming back to your problem, glmnet works only on matrices so please be sure to get all your data in the matrix form. Create two separate matrices one for the predictors (X) and another for the response (Y). Run the following code:

library(glmnet)
loReg <- glmnet(x=X, y=Y, family = "binomial", lower.limits = 0, lambda = 0, standardize=TRUE)

The above line will create a logistic model with penalizing coefficient equal to zero (which is what you want). Since the lower limit of all of your variables is the same (i.e. zero), setting lower.limits=0 will do the job.

To predict new observation: Suppose you want to predict m new observations. Get these observations in an mxp matrix, where p is the number of predictors. Let this matrix of new observations be newX.

predict(loReg, newx = newX, type="response")

Hope this would help.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.