Demand forecasting: what is the proper validation strategy to objectively compare a new algorithm with the one already used in the system?

Question

Although the question is related to any problem related to demand forecasting, I would like to describe our framework:

The framework:

We have different products being sold for a short period of time (maximum 3 weeks) once a year. Some products may have no sales history - new products. Since this is not a classical time series forecasting we decided to use ML (Machine Learning) approach by estimating the future demand based on the history of sales using different features like price, name of a product (text based features), store information, store location, etc. The assumption is that the model will learn complex interactions between all features and will forecast sales for the next year. Second assumption is that our sales data is the best we can get to estimate the future demand. So sales=demand.

Current situation

Currently the demand forecasting is performed by a human expert. The intention is to support his decisions or even replace a human judgement with model-based forecasts

Validation Problem:

The model building process is performed as usual in ML by training a model on a training set and validating ML performance on a hold-out set. In any ML-related problem it wouldn't be an issue to compare any algorithm on a hold-out set and pick the one that has a minimum error. However in demand forecasting the issue is that we compare the ML forecasts (order quantity) with the results of a human prediction (sales) and not with a human prediction itself (order quantity).

I will bring 2 examples to stress the difference:

Usual Case: Let's say we want to compare two algorithms that predict stock price for a company X next day. One algorithm predicts 1000, another one predicts 1500. All we need to do is to test it on a hold-out set or just wait until tomorrow and check what algorithm was close to the prediction

Our Case: We want to compare who predicts demand better. A human expert says we should order 1000 units for a product, ML predicts 500 units. Let's also say that the theoretical demand is 800 units. Now there are two alternative "futures" (but only one of them is resulted in the hold-out set):

1. If we order according to the human expert (which actually happens right now), we order 1000 and sell 800 (and this is what we will see in our hold-out set). If we compare the prediction error of ML, it is higher in absolute terms abs(500-800) > abs(1000-800).
2. If we order according to the ML prediction, we order 500 and sell 500 while theoretical demand is 800. If we compare the prediction error of ML now, it is 0 (500-500), while the error of the human expert is 500: abs(1000 - 500).

This clearly suggests that the comparison against the actual sales is not objective even if we compare it on a hold-out set.

The biggest question is against what entity can we compare a human expert and ML in order to decide who makes better predictions? My logic says that it should be some theoretical demand calculated on a training set for each product. For example it can be the average of last X sales or just last sale for same product as is commonly done in time series forecasting when building a baseline statistics. In this case the average sales can be compared to the prediction of a human expert and of ML algorithm and pick the one whose deviation from average is the least. But I am not completely sure if this is the correct way...

Answer 1

The key problem is that your two "systems" are not forecasting the same thing.

• Your human expert is forecasting a high quantile of the future demand distribution, to account for variance. Essentially, he is already including the safety amount.
• Your ML system is giving the forecast that optimizes the error measure it is trying to optimize. You do not write which one that is. In such a case, it is quite probably maximizing a (log-)likelihood, or minimizing in-sample MSE. Which would amount to its predicting the expectation of the future demand.

Of course the two values are different. (Technically speaking, you are eliciting two different functionals of the future demand distribution.)

Now, I would argue that the first thing to do is to either post-process your ML predictions, or to change the way they are calculated in the first place. After all, if you only stock up to expected demand, you will run out of stock often, and your customers will be unhappy. So you will need safety stock - as your human is implicitly ensuring. You can do this by either using your ML's expectation forecast and then adding a safety stock on top, or by changing the ML's objective function to also forecast a high quantile. Either way will give you a quantile forecast from the ML system that aims at the same thing your human expert is aiming at. Now your forecasts are comparable.

To do so, you will need to decide on what quantile you should be aiming at. You may be able to get some information out of your human expert. Is he trying to satisfy 90% of demand, or 95%, or 99%? Note that in your line of business, any product left over at the end of its season is likely a bit of a problem, so you may consciously aim at lower service levels to ensure you don't often have leftovers.

(Note that there are multiple definitions of "service levels". How these map to optimal quantile forecasts is a tricky problem that depends on the actual distribution of your demand.)

Finally, assuming you have true quantile forecasts from your human and your ML expert, both of which aim at the same level, say $\alpha=90<%$. Now you can assess these using a hinge loss of the type that is also used in quantile regression:

$$ L(y, \hat{y}) := \rho_\alpha(y-\hat{y}), \text{ where }\rho_\alpha(x)= \begin{cases} \alpha x, & x\geq 0 \\ (\alpha-1)x, & x<0. \end{cases}$$

for quantile forecasts $\hat{y}$ and actuals $y$. I very much recommend Gneiting (2011, IJF) on this.