I have a neural network with a softmax at the end.
Something like this:
def forward(self, x) x = self.conv(x) x = self.channel_transform_layer(x) output = self.softmax(x) return output
I would like the maximum value of the logits to be
p being between 0 and 1, say 0.7). I'm working on a task where an output greater than
p does not make sense, so I want to constrain all the logits to be between 0 and
taking a concrete example with pytorch:
import torch softmax = torch.nn.functional.softmax softmax(torch.Tensor([1,1,5])) # => tensor([0.0177, 0.0177, 0.9647]) # I want to define constrained_softmax such that constrained_softmax(torch.Tensor([1,1,5]), p=0.7) # => tensor([0.175, 0.175, 0.65]) # those are approximate values to get the idea. Importantly, all # values should sum to 1 and the max logit should be < p
I tried tweaking the softmax without success. I also tried to apply the softmax multiple times and scaling it, but I don't end up with the desired result.