# Demand forecasting: what is the proper validation strategy to objectively compare a new algorithm with the one already used in the system?

Although the question is related to any problem related to demand forecasting, I would like to describe our framework:

The framework:

We have different products being sold for a short period of time (maximum 3 weeks) once a year. Some products may have no sales history - new products. Since this is not a classical time series forecasting we decided to use ML (Machine Learning) approach by estimating the future demand based on the history of sales using different features like price, name of a product (text based features), store information, store location, etc. The assumption is that the model will learn complex interactions between all features and will forecast sales for the next year. Second assumption is that our sales data is the best we can get to estimate the future demand. So sales=demand.

Current situation

Currently the demand forecasting is performed by a human expert. The intention is to support his decisions or even replace a human judgement with model-based forecasts

Validation Problem:

The model building process is performed as usual in ML by training a model on a training set and validating ML performance on a hold-out set. In any ML-related problem it wouldn't be an issue to compare any algorithm on a hold-out set and pick the one that has a minimum error. However in demand forecasting the issue is that we compare the ML forecasts (order quantity) with the results of a human prediction (sales) and not with a human prediction itself (order quantity).

I will bring 2 examples to stress the difference:

Usual Case: Let's say we want to compare two algorithms that predict stock price for a company X next day. One algorithm predicts 1000, another one predicts 1500. All we need to do is to test it on a hold-out set or just wait until tomorrow and check what algorithm was close to the prediction

Our Case: We want to compare who predicts demand better. A human expert says we should order 1000 units for a product, ML predicts 500 units. Let's also say that the theoretical demand is 800 units. Now there are two alternative "futures" (but only one of them is resulted in the hold-out set):

1. If we order according to the human expert (which actually happens right now), we order 1000 and sell 800 (and this is what we will see in our hold-out set). If we compare the prediction error of ML, it is higher in absolute terms abs(500-800) > abs(1000-800).
2. If we order according to the ML prediction, we order 500 and sell 500 while theoretical demand is 800. If we compare the prediction error of ML now, it is 0 (500-500), while the error of the human expert is 500: abs(1000 - 500).

This clearly suggests that the comparison against the actual sales is not objective even if we compare it on a hold-out set.

The biggest question is against what entity can we compare a human expert and ML in order to decide who makes better predictions? My logic says that it should be some theoretical demand calculated on a training set for each product. For example it can be the average of last X sales or just last sale for same product as is commonly done in time series forecasting when building a baseline statistics. In this case the average sales can be compared to the prediction of a human expert and of ML algorithm and pick the one whose deviation from average is the least. But I am not completely sure if this is the correct way...

The key problem is that your two "systems" are not forecasting the same thing.

• Your human expert is forecasting a high quantile of the future demand distribution, to account for variance. Essentially, he is already including the safety amount.
• Your ML system is giving the forecast that optimizes the error measure it is trying to optimize. You do not write which one that is. In such a case, it is quite probably maximizing a (log-)likelihood, or minimizing in-sample MSE. Which would amount to its predicting the expectation of the future demand.

Of course the two values are different. (Technically speaking, you are eliciting two different functionals of the future demand distribution.)

Now, I would argue that the first thing to do is to either post-process your ML predictions, or to change the way they are calculated in the first place. After all, if you only stock up to expected demand, you will run out of stock often, and your customers will be unhappy. So you will need safety stock - as your human is implicitly ensuring. You can do this by either using your ML's expectation forecast and then adding a safety stock on top, or by changing the ML's objective function to also forecast a high quantile. Either way will give you a quantile forecast from the ML system that aims at the same thing your human expert is aiming at. Now your forecasts are comparable.

To do so, you will need to decide on what quantile you should be aiming at. You may be able to get some information out of your human expert. Is he trying to satisfy 90% of demand, or 95%, or 99%? Note that in your line of business, any product left over at the end of its season is likely a bit of a problem, so you may consciously aim at lower service levels to ensure you don't often have leftovers.

(Note that there are multiple definitions of "service levels". How these map to optimal quantile forecasts is a tricky problem that depends on the actual distribution of your demand.)

Finally, assuming you have true quantile forecasts from your human and your ML expert, both of which aim at the same level, say $$\alpha=90<%$$. Now you can assess these using a hinge loss of the type that is also used in quantile regression:

$$L(y, \hat{y}) := \rho_\alpha(y-\hat{y}), \text{ where }\rho_\alpha(x)= \begin{cases} \alpha x, & x\geq 0 \\ (\alpha-1)x, & x<0. \end{cases}$$

for quantile forecasts $$\hat{y}$$ and actuals $$y$$. I very much recommend Gneiting (2011, IJF) on this.

• To the question what we optimize. Since we are using a gradient boosting algorithm we have plenty of loss functions to test: MSE, poisson, MAPE. The stopping criteria can be for example RMSE or MAPE. Final comparison is performed using WMAPE weighted by price of a product Mar 5, 2021 at 14:06
• You may be interested in this recent thread. Mar 5, 2021 at 14:09
• I will add one more layer of complexity to Stephan's excellent explanation. In the retail context (I've seen this happen a couple of times) you might actually make worse decisions (more out of stock events, or more excess inventory) because your forecast accuracy improved. How does this happen? Because your previous replenishment policy (the various parameters you used to decide when and how much to reorder given a certain forecast) was calibrated for your previous forecast accuracies, and by improving the accuracy of the forecasts you're "confusing" the system so to speak. Mar 13, 2021 at 9:28
• "Finally, assuming you have true quantile forecasts from your human expert" - what should we do in case we don't have information about the true quantiles? I used the following approach but I am not sure if it was the correct way. I trained my models on different quantiles. Then I used Hinge Loss to compare predictions with the human expert until I reached almost equal scores for both. Mar 16, 2021 at 13:33
• You mean that you don't know which quantile the expert is shooting for? If so then your approach in your last comment sounds reasonable. (But of course, training a model until it replicates the target your human is aiming for, then assessing which one is better at reaching that self-imposed target, makes little sense. Better to first think about which target makes sense in terms of costs of over- and undersupply, then try to achieve that target. Changing the target is easier for machines than for humans.) Mar 17, 2021 at 6:43