2
$\begingroup$

I have a neural network with three output neurons $X_1$, $X_2$ and $X_3$ with output range in [-1, 1].

Sample Neural Network Images

I have many training data split in 80:20 ratio between training & testing sets. While looking at the output of the neural network, I noticed a big thing that the neural network haven't learned.

There is a restriction in the output as $X_1 < X_2 < X_3$.

I found that only with training, I can't achieve this restriction. Is there a recommended way theoretically or using Keras library to tell the neural network about this restriction.

$\endgroup$

1 Answer 1

2
$\begingroup$

You can enforce the constraint with an additional transformation. Suppose we have $x_i = \sigma(AH + b)$ where $\sigma$ is the sigmoid activation and $A, b$ are the weights and biases.

We can introduce a fixed matrix $W = \begin{bmatrix}1 & 0 & 0 \\ 1 & 1 & 0 \\ 1 & 1 & 1\end{bmatrix}$, which is not updated during back-propagation (that is, it is non-trainable). We know that $Wx$ will satisfy the constraint because $x_i$ are all positive, so we have $0 < x_1 < x_1 + x_2 < x_1 + x_2 + x_3 < 3$.

If you need the result to be in $[-1,1]$, then you can just rescale and shift: $\frac{2}{3}Wx - 1$. The choice of $\frac{2}{3}$ means that the largest value possible is 2, and the smallest possible value remains 0. Shifting by $-1$ moves the interval from $[0,2]$ to $[-1,1]$.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.