Can I enforce monotonically increasing neural net outputs (min, mean, max)? Hi I'm using a DL model (TensorFlow) to predict daily minimum, mean, and maximum values of a target dataset. I was thinking that the model would have 3 outputs for each day, (min, mean, max).
Is there a clean way to enforce the correct order of these (i.e., min<mean<max)? I can add a penalty to encourage the model to train that way, but that seems like a bit of a work-around.
 A: It may be infeasible to formally check the KKT conditions for your optimization problem, however you can still try encoding your inequality constraints as if the conditions hold. Then it is a matter of checking whether the training behaves nicely in practice.
If you're unfamiliar with encoding constraints into an objective function in the way that KKT states, see Lagrange multipliers for a first example with equality constraints. Then I recommend you look at encoding inequality constraints (see example) in a similar fashion.
Once you have the mathematical expression in hand, you'll need to implement it in Tensorflow. You can build your own loss function class by inheriting from the tf.keras.losses.Loss base class.
A: As mentioned in above discussion with @Galen:
Conjecture This might be achieved with custom recurrent layer. We could provide a box plot values as an output, i.e., Five number summary as a monotonic output. Though, internals of recurrent layer is a design choice, see Define custom LSTM Cell in Keras?. Here, output of LSTM cell will be our monotonic summary function. This approach guarantees the inequality.
A: Two techniques: penalty and variable transformation.
penalty
build one model with these three outputs, then modify/customize the loss function during its estimation by adding the penalty for violation of the assumption. this will not guarantee the inequalities but will make them very unlikely.
You can simply add $-\lambda[\min(y_2-y_1,0)+\min(y_3-y_2,0)]$ where $\lambda$ is a hyper parameter reflecting how badly you want to enforce the conditions, and $y_1,y_2,y_3$ are you min, mean and max outputs. I’m using ReLU function here, but you can use any strictly positive function.
variable transformation
I use this technique in similar situations. Here's how it goes. Create new variables: $$y'_1=y_2\\y'_2=\ln(y_2-y_1)\\y'_3=\ln(y_3-y_2)$$
Now you can fit the unconstrained model to new variables, then reconstruct the outputs as
$$y_2=y'_2\\y_1=y_2-e^{y'_2}\\y_3=y_2+e^{y'_3}$$
The outputs will be guaranteed to have the required conditions.
There are variations, e.g. you can transform min, mean and max into mean, range and mean/range etc which can be more stable. You can replace exponent with any strictly positive function such as ReLU, as it is noted in a comments.
This may look like a better technique, but it has its own issues. The main one is that fitting to logarithm can produce very wild forecasts. It's one reason why you should not transform the mean itself, and only min and max are transformed to distances from the mean. This way at least we may get reasonable mean forecast, and maybe crazy min and max, which are expected to be lousy anyways.
related issue
Another thing to be aware of is that usually mean forecast should be expected to have lower variance than min and max. Therefore, you may make some accommodations in your loss function to allow min and max have larger forecast error than mean.
A: I would suggest you are better off investigating quantile regression.
see eg
https://towardsdatascience.com/deep-quantile-regression-in-tensorflow-1dbc792fe597.
What I believe you want is to predict the mean and limits of your distribution. at each x value.  using the minimum and maximum as targets will not give you what you want. instead you will have the expected value of the minimum/maximum at each input (this is what MSE produces).
eg input is x




id
x
y_mean
y_max




1
1
1
1


2
1
1
3


3
1
1
5




then the prediction for y_max for input x=1 will be 3, not 5.
if instead you predict the quantiles you will get eg 10 percentile, 50 percentile and 90 percentile of your distribution.
(an alternative simpler way is to continue with mse,but predict the mean and squared error.  that gives you the variance and you can eg estimate the percentiles making a normal approximation)
A: Unless I have misunderstood the question, here's a simple approach.
For outputs a b and c, you could enforce positive values for the second and third outputs via a suitable activation function, and then train to find values a b and c, such that a is the min, a + b is the mean, and a + b + c is the max value.
A: Why not applying cumsum layer at the very end of your model?
By doing that, the model will predict the difference between t-1 and t on the layer n-1.
Something like:
tf.math.cumsum(y_features, axis=time_series_axis_index)`

