Although the question is related to any problem related to demand forecasting, I would like to describe our framework:
We have different products being sold for a short period of time (maximum 3 weeks) once a year. Some products may have no sales history - new products. Since this is not a classical time series forecasting we decided to use ML (Machine Learning) approach by estimating the future demand based on the history of sales using different features like price, name of a product (text based features), store information, store location, etc. The assumption is that the model will learn complex interactions between all features and will forecast sales for the next year. Second assumption is that our sales data is the best we can get to estimate the future demand. So sales=demand.
Currently the demand forecasting is performed by a human expert. The intention is to support his decisions or even replace a human judgement with model-based forecasts
The model building process is performed as usual in ML by training a model on a training set and validating ML performance on a hold-out set. In any ML-related problem it wouldn't be an issue to compare any algorithm on a hold-out set and pick the one that has a minimum error. However in demand forecasting the issue is that we compare the ML forecasts (order quantity) with the results of a human prediction (sales) and not with a human prediction itself (order quantity).
I will bring 2 examples to stress the difference:
Usual Case: Let's say we want to compare two algorithms that predict stock price for a company X next day. One algorithm predicts 1000, another one predicts 1500. All we need to do is to test it on a hold-out set or just wait until tomorrow and check what algorithm was close to the prediction
Our Case: We want to compare who predicts demand better. A human expert says we should order 1000 units for a product, ML predicts 500 units. Let's also say that the theoretical demand is 800 units. Now there are two alternative "futures" (but only one of them is resulted in the hold-out set):
- If we order according to the human expert (which actually happens right now), we order 1000 and sell 800 (and this is what we will see in our hold-out set). If we compare the prediction error of ML, it is higher in absolute terms abs(500-800) > abs(1000-800).
- If we order according to the ML prediction, we order 500 and sell 500 while theoretical demand is 800. If we compare the prediction error of ML now, it is 0 (500-500), while the error of the human expert is 500: abs(1000 - 500).
This clearly suggests that the comparison against the actual sales is not objective even if we compare it on a hold-out set.
The biggest question is against what entity can we compare a human expert and ML in order to decide who makes better predictions? My logic says that it should be some theoretical demand calculated on a training set for each product. For example it can be the average of last X sales or just last sale for same product as is commonly done in time series forecasting when building a baseline statistics. In this case the average sales can be compared to the prediction of a human expert and of ML algorithm and pick the one whose deviation from average is the least. But I am not completely sure if this is the correct way...