2
$\begingroup$

My goal to implement a neural net which outputs a range of values with an additional restriction: based on an input value selected among the features, certain output values will be forbidden.

For example, suppose an input value is in the range 1 to 100. The output would be forced to be greater than the input value but less than 100. Another example would be to restrict output to the input +-10.

Would simply adding +Inf to output values in the forbidden range accomplish this?

$\endgroup$

2 Answers 2

2
$\begingroup$

One way to code such constraints is using something akin to an SVM hinge loss. For $f_i(x)$ be greater than $x_i$, this is morally equivalent to adding a loss term $\alpha_i\max(x_i-f_i(x),0)$, where $\alpha_i$ is a hyperparameter. To make this more stable, typicall you'd add a separation hyperparameter: $\alpha_i\max(x_i-f_i(x)+\beta_i,0)$. You can add strict upper/lower bounds in a similar way. In general this is a soft constraint because you're only enticing your model to try and make the output bigger than the input, but sometimes this will be false. By making $\alpha_i$ very large, it will do its best to avoid these issues. Theoretically if $\alpha_i$ is big enough, and your neural network is big enough, it will find a way to satisfy the constraints exactly. But this heavily risks over-regularizing the network (are you worried more about accuracy or escaping your constraints?).

If you want to strictly force your output to be bigger than the input, you might need an adhoc activation function in your last layer, that's also connected to your inputs. So for example if $z_i$ are the outputs of the prior layer, maybe something like:

$$a(Wz+x)$$

where the weight matrix $W$ is restricted to have only positive coefficients, and $a$ is monotonically increasing. Packages like keras have options to make such weight restrictions. If you want to further restrict the range of $a$, you can play around with the activation functions range. For example if $a(x)$ is a sigmoid, its values are in $(0,1)$, but $99a(x)+1$ takes values in $(1,100)$.

$\endgroup$
1
  • $\begingroup$ Thank you for the detailed response! I wish to strictly force the output to be bigger than the input, so will try the weight restriction. Is it difficult to implement this manually, without using a package? $\endgroup$
    – FFT
    Commented May 5, 2017 at 17:30
0
$\begingroup$

To restrict your output to a simple inequality as x>=0, you could use a relu activation in your last layer. If the lower limit is different from zero, you could tweak it in the custom loss. For an interval constraint, you could use a similar idea and the sigmoid function.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.