I have a ML model (a NN in the specific but I don't think it's important for the purpose of my question) that is doing pretty decent at his job, which is predicting the demand of a certain substance X in the next day (on day n I have to predict the demand of X for day n+1). The overall prediction error is satisfying, the prediction error on average is below 2% with respect with the true demand of the following day, so in general I will be pretty satisfied.
My problem is that, for the particular problem I am facing, I cannot make a prediction with an error above a certain threshold, say 10%, because if that happens I have to pay a penalty. Everything below 10% error is fine and completely equal, there is no gain in predicting the demand with an error of 0.1% or 9%, the only important thing is that the prediction error remain under 10%.
So, just to be clear, I would prefer a model that always predict with an error of 9.9%, instead of one that every time is below 1% but one single time goes to 11%, because my ultimate goal is to never pay a penalty (or at least, as few as possible).
My initial idea, was to add a penalisation to my loss function during the training, in order to heavily penalise (and here also there is the problem of how much to penalize) every error that falls above 10% weights like 100 times more than an error below 10%, which will weight 1/100.
I would love to know if my idea makes sense, if there are already some know techniques available online or any feedback.
I would like to enlarge my question and ask a more general question.
Suppose that I have trained a model with a certain loss function, one of the classical one like L1 or L2.
But my problem is tricky since depending on the amount of percentage error that I make I pay a certain penalty, lets say something like:
- [0%, 5%] I pay nothing
- [5%, 10%] I pay 10
- [10%, 25%] I pay 50
- [25%, +inf] I pay 200
How should I adjust my model? The possible solutions that comes to my mind are
- Change completely the loss function and create my custom loss function that replicate exactly my penalty payments
- Change the original loss function by adding some weights depending on the intervals above