4
$\begingroup$

I am solving a binary classification task, and I need my logistic regression's learned weights to be all positive. This is my current classifier implemented in pytorch :

class LogisticRegression(torch.nn.Module):
    def __init__(self, input_dim, output_dim):
        super(LogisticRegression, self).__init__()
        self.linear = torch.nn.Linear(input_dim, output_dim)

    def forward(self, x):
        outputs = self.linear(x)
        return outputs

So how should I change the code to force the weights to be always positive?

EDIT : Keras has an option that can cause the weights of the model to be non negative :

tf.keras.constraints.NonNeg()

https://keras.io/api/layers/constraints/

basically my question is : what is the equivalent of this in pytorch?

$\endgroup$
3
  • 1
    $\begingroup$ Negating the variables with negative weights would work fine. Seriously, why do you want this? $\endgroup$
    – Nick Cox
    Nov 13, 2020 at 9:40
  • $\begingroup$ @NickCox How can i do this in pytorch? which part of code should i change? $\endgroup$
    – OneAndOnly
    Nov 13, 2020 at 10:04
  • $\begingroup$ No idea, sorry. Fine software, no doubt, that I have never used. But you missed that it is a frivolous suggestion. The question is on all fours with: I get a negative coefficient for a regression on #coding errors with experience. How do I get a positive coefficient? $\endgroup$
    – Nick Cox
    Nov 13, 2020 at 10:16

1 Answer 1

4
$\begingroup$

Alhtough I cannot think of a reasonable use case, technically it is simple. You can make your own linear layer that will use the absolute value of the weight (or any function that will ensure the weights are positive) in the forward function.

import torch
import torch.nn as nn

class PosLinear(nn.Module):
    def __init__(self, in_dim, out_dim):
        super(PosLinear, self).__init__()
        self.weight = nn.Parameter(torch.randn((in_dim, out_dim)))
        self.bias = nn.Parameter(torch.zeros((out_dim,)))
        
    def forward(self, x):
        return torch.matmul(x, torch.abs(self.weight)) + self.bias

Note that if you want to get effective weights, you need to apply the same function as in the foward method.

$\endgroup$
5
  • $\begingroup$ Thanks for answer, in an answer somewhere else, it was suggested to use non negative logistic regression to have non negative weights, is this true? i googled but couldn't find anything useful regarding a "non negative" logistic regression $\endgroup$
    – OneAndOnly
    Nov 13, 2020 at 10:42
  • 1
    $\begingroup$ Never heard of such a thing. $\endgroup$
    – Jindřich
    Nov 13, 2020 at 10:55
  • $\begingroup$ Ok i found something that does this in keras : tf.keras.constraints.NonNeg(), so is the method you suggested the same as this? if not, what is the equivalent of this in pytorch? $\endgroup$
    – OneAndOnly
    Nov 13, 2020 at 11:51
  • 3
    $\begingroup$ @Jindřich With weights that sum to 1, this is one way to find the optimal weighted average of some predicted probabilities (=model blending). I'd agree though that situations where you want coefficients >0, but not sum(coefficients)=1 seem unusual. You'd have to have a situation where some minimum probabilitiy is always guranteed, but some factors could increase it - I can't think of many problems like that. $\endgroup$
    – Björn
    Nov 13, 2020 at 11:57
  • 2
    $\begingroup$ also is it really a good idea to take the absolute of weights to remove negatives, instead of converting negatives to 0? $\endgroup$
    – OneAndOnly
    Nov 13, 2020 at 15:04

Not the answer you're looking for? Browse other questions tagged or ask your own question.