Constrain a Neural Network to be monotonic? Is there a way or theorem that allows me to constrain a function $N(x)$, where $N(x)$ is a feedforward mlp such that it will always be a monotonic function. The simplest way is of course to use a penalty method, but this does not 100% guarantee, I am looking to see if there is a function or transformation that can be applied to $N(x)$ that constrains its form to being monotonic. 
The only way I can currently think of is by making all weights $w_{i,j,k} >0$ and that the activation function $f(a)$ are monotonic and non-negative (using $f(a) = \frac{tanh(a)+1}{2}$) and having a zero bias.
 A: You may want to have a look at "Unconstrained Monotonic Neural Networks".
The basic idea is to construct a neural network that forces the output to be positive. The integral of the output from that neural network is the final output that forms a monotonic function.
The paper describes how to train such a neural network. That is to say how to get the derivatives for the parameters with the added integral.
Here is the link to the paper: https://arxiv.org/pdf/1908.05164.pdf
A: Here's an example of an early publication in this vein. 
Joseph Sill. "Monotonic Networks". California Institute of Technology. 1998.

Monotonicity is a constraint which arises in many application domains. We present a machine learning model, the monotonic network, for which monotonicity can be enforced exactly, i.e., by virtue of functional form. A straightforward method for implementing and training a monotonic network is described. Monotonic networks are proven to be universal approximators of continuous, differentiable monotonic functions. We apply monotonic networks to a real-world task in corporate bond rating prediction and compare them to other approaches. 

