Penalising Error above a certain Threshold

Question

I have a ML model (a NN in the specific but I don't think it's important for the purpose of my question) that is doing pretty decent at his job, which is predicting the demand of a certain substance X in the next day (on day n I have to predict the demand of X for day n+1). The overall prediction error is satisfying, the prediction error on average is below 2% with respect with the true demand of the following day, so in general I will be pretty satisfied.

My problem is that, for the particular problem I am facing, I cannot make a prediction with an error above a certain threshold, say 10%, because if that happens I have to pay a penalty. Everything below 10% error is fine and completely equal, there is no gain in predicting the demand with an error of 0.1% or 9%, the only important thing is that the prediction error remain under 10%.

So, just to be clear, I would prefer a model that always predict with an error of 9.9%, instead of one that every time is below 1% but one single time goes to 11%, because my ultimate goal is to never pay a penalty (or at least, as few as possible).

My initial idea, was to add a penalisation to my loss function during the training, in order to heavily penalise (and here also there is the problem of how much to penalize) every error that falls above 10% weights like 100 times more than an error below 10%, which will weight 1/100.

I would love to know if my idea makes sense, if there are already some know techniques available online or any feedback.

Edit

I would like to enlarge my question and ask a more general question.

Suppose that I have trained a model with a certain loss function, one of the classical one like L1 or L2.

But my problem is tricky since depending on the amount of percentage error that I make I pay a certain penalty, lets say something like:

• [0%, 5%] I pay nothing
• [5%, 10%] I pay 10
• [10%, 25%] I pay 50
• [25%, +inf] I pay 200

How should I adjust my model? The possible solutions that comes to my mind are

• Change completely the loss function and create my custom loss function that replicate exactly my penalty payments
• Change the original loss function by adding some weights depending on the intervals above

Answer 1

In general, it is best if the loss function in fitting a model is as close as possible to the real cost your predictions would incur. Yes, these costs are often very hard to operationalize. I would argue that this is a Good Thing: once you realize it is hard to pin an exact real-life cost to your model's output, you understand that it may not be worthwhile trying to tease the last bit of performance out of your model using your pretend-cost.

I am still rather sceptical that the only cost that matters is the penalty. I'll accept that the penalty is the same whenever the percentage error is exceeded, whether by 11% or by 50%... but I find it hard to believe that there are truly no other costs at all.

But if this is a reasonable modeling assumption, then you can simply use a custom loss function that is zero if the prediction is less that 10% from the actual, and one otherwise.

I went and simulated this a bit, similar to finding the optimal point prediction if your actuals are conditionally gamma or conditionally lognormal under various loss functions. Playing around a bit with this loss function, it often turns out that the optimal prediction is pretty close to the expectation, at least for the parameters I fed into my simulations. So you might start out with a straightforward model trained on squared errors (which elicits the expectation) - this might already give pretty good results. For instance, the plot below shows what loss you can expect for given forecasts if your data is conditionally lognormal with log-mean 1 and log-SD 0.2, and the vertical line gives the expectation, which is pretty close to the loss-minimizing forecast (R code at bottom):

(These simulations also show that there is some limit to the possible accuracy. In the plot above, you won't get below incurring that penalty in 60% of cases. If there is a lot of residual noise, then no matter what you forecast, you have a good chance of incurring that penalty often. There is simply a limit to forecastability, which depends on the process we are forecasting. See How to know that your machine learning problem is hopeless? and Ways to increase forecast accuracy.)

R code:

set.seed(1) # for reproducibility
# sims <- rgamma(1e4,1.4,1.4)
# sims <- rnorm(1e4,mean=10,sd=1)
sims <- rlnorm(1e4,meanlog=1,sdlog=0.2)

sims <- sims[sims>0]    # in case we simulate something below zero

forecasts <- seq(min(sims),max(sims),by=0.1)
error_function <- Vectorize(function(fcst,actual) {
    if ( abs(fcst-actual) <= 0.1*actual ) 0 else 1
},"actual")

errors <- sapply(forecasts,function(xx)mean(error_function(xx,sims)))
plot(forecasts,errors,type="o",pch=19,las=1)
abline(v=mean(sims))