# Penalising Error above a certain Threshold

I have a ML model (a NN in the specific but I don't think it's important for the purpose of my question) that is doing pretty decent at his job, which is predicting the demand of a certain substance X in the next day (on day n I have to predict the demand of X for day n+1). The overall prediction error is satisfying, the prediction error on average is below 2% with respect with the true demand of the following day, so in general I will be pretty satisfied.

My problem is that, for the particular problem I am facing, I cannot make a prediction with an error above a certain threshold, say 10%, because if that happens I have to pay a penalty. Everything below 10% error is fine and completely equal, there is no gain in predicting the demand with an error of 0.1% or 9%, the only important thing is that the prediction error remain under 10%.

So, just to be clear, I would prefer a model that always predict with an error of 9.9%, instead of one that every time is below 1% but one single time goes to 11%, because my ultimate goal is to never pay a penalty (or at least, as few as possible).

My initial idea, was to add a penalisation to my loss function during the training, in order to heavily penalise (and here also there is the problem of how much to penalize) every error that falls above 10% weights like 100 times more than an error below 10%, which will weight 1/100.

I would love to know if my idea makes sense, if there are already some know techniques available online or any feedback.

Edit

I would like to enlarge my question and ask a more general question.

Suppose that I have trained a model with a certain loss function, one of the classical one like L1 or L2.

But my problem is tricky since depending on the amount of percentage error that I make I pay a certain penalty, lets say something like:

• [0%, 5%] I pay nothing
• [5%, 10%] I pay 10
• [10%, 25%] I pay 50
• [25%, +inf] I pay 200

How should I adjust my model? The possible solutions that comes to my mind are

• Change completely the loss function and create my custom loss function that replicate exactly my penalty payments
• Change the original loss function by adding some weights depending on the intervals above
• Is the penalty the same if your error is 50% as if it is 11%? That will make a difference to what cost function you want. Also any other cost drivers. I would find it hard to believe there are absolutely no other cost or profit considerations than just not exceeding the threshold (because if so, here is your penalty-minimizing solution: simply go out of business, then you will never pay a penalty). Getting the "best" cost function will likely depend on many other influences, which you will need to understand first. Ideally, the cost function should reflect the true costs you incur. Jan 19 at 10:59
• I have edited my question to make a more general problem, thanks to your comment, that made me realize that makes obviously more sense to pay different amount depending on the amount of the error Jan 20 at 8:41
• You would need to figure out how the penalty compares to any other costs, or the $L^1$/$L^2$ loss. If the $L^p$ loss of any prediction is irrelevant, you can simply use a loss function that is stepwise per your penalty. Otherwise, it makes sense to use a combined loss. Per my answer, what I am advocating getting a good a grip on your actual loss as possible, and then including that as a loss function in fitting. (Per Richard's comments to my answer, you could also go for full density predictions and then extract the optimal point prediction from that using your cost function.) Jan 20 at 9:26

In general, it is best if the loss function in fitting a model is as close as possible to the real cost your predictions would incur. Yes, these costs are often very hard to operationalize. I would argue that this is a Good Thing: once you realize it is hard to pin an exact real-life cost to your model's output, you understand that it may not be worthwhile trying to tease the last bit of performance out of your model using your pretend-cost.

I am still rather sceptical that the only cost that matters is the penalty. I'll accept that the penalty is the same whenever the percentage error is exceeded, whether by 11% or by 50%... but I find it hard to believe that there are truly no other costs at all.

But if this is a reasonable modeling assumption, then you can simply use a custom loss function that is zero if the prediction is less that 10% from the actual, and one otherwise.

I went and simulated this a bit, similar to finding the optimal point prediction if your actuals are conditionally gamma or conditionally lognormal under various loss functions. Playing around a bit with this loss function, it often turns out that the optimal prediction is pretty close to the expectation, at least for the parameters I fed into my simulations. So you might start out with a straightforward model trained on squared errors (which elicits the expectation) - this might already give pretty good results. For instance, the plot below shows what loss you can expect for given forecasts if your data is conditionally lognormal with log-mean 1 and log-SD 0.2, and the vertical line gives the expectation, which is pretty close to the loss-minimizing forecast (R code at bottom): (These simulations also show that there is some limit to the possible accuracy. In the plot above, you won't get below incurring that penalty in 60% of cases. If there is a lot of residual noise, then no matter what you forecast, you have a good chance of incurring that penalty often. There is simply a limit to forecastability, which depends on the process we are forecasting. See How to know that your machine learning problem is hopeless? and Ways to increase forecast accuracy.)

R code:

set.seed(1) # for reproducibility
# sims <- rgamma(1e4,1.4,1.4)
# sims <- rnorm(1e4,mean=10,sd=1)
sims <- rlnorm(1e4,meanlog=1,sdlog=0.2)

sims <- sims[sims>0]    # in case we simulate something below zero

forecasts <- seq(min(sims),max(sims),by=0.1)
error_function <- Vectorize(function(fcst,actual) {
if ( abs(fcst-actual) <= 0.1*actual ) 0 else 1
},"actual")

errors <- sapply(forecasts,function(xx)mean(error_function(xx,sims)))
plot(forecasts,errors,type="o",pch=19,las=1)
abline(v=mean(sims))

• Regarding In general, it is best if the loss function in fitting a model is as close as possible to the real cost your predictions would incur.: what about the famous example of targetting the median but estimating the mean of a Gaussian random variable? It illustrates that choosing the training loss equal to evaluation loss may be a bad idea. Why not do maximum likelihood or Bayesian estimation first and then derive an optimal point forecast based on the evaluation loss function? Jan 19 at 17:44
• @RichardHardy: if we know or can reasonably enough assume our conditional distributions to be Gaussian, sure. (Just as I suggested starting out with an expectation prediction.) And yes, of course we could try modeling a predictive density and then derive the optimal point forecast from that. OP sounds like they have a NN architecture where they can easily switch out a point prediction loss function - going ML or full Bayesian may require larger changes to the entire setup. Jan 19 at 18:33
• I think we would need to assume some distribution, not necessarily Gaussian. Or we could target the conditional mean and estimate the distribution around it nonparametrically. But in any case, my basic point is about in general. Jan 19 at 18:48
• @RichardHardy: I don't see why we need to assume a specific (conditional) distribution. Compare quantile regression: conditional quantiles are directly estimated by minimizing the pinball loss. No explicit consideration of a particular distribution. I have to admit I don't quite grasp what you mean by "in general", can you elucidate? Jan 19 at 19:05
• Well, you suggested to assume a Normal distribution, so I tried to say this is an unnecessary restriction. Now you have relaxed it even further, but then we are not using maximum likelihood or Bayesian estimation anymore. And that is OK, too. Regarding In general, I disagree with that statement. In some cases, it is so, but not in general. Jan 19 at 19:10