Somewhat new to the forecasting area. I am trying to evaluate whether the statistical forecasts are better than manually generated forecasts in one of our used cases. I have 1000s of customers who consume our products daily. A broad question that comes to my mind is: How will you decide which methods works better on this problem of forecasting daily consumption? So, I conducted a man. vs. machine. exercise.
- Man = Manually generated forecasts
- Machine = ARIMA, Exp. Smoothing, TBATS etc.
I split the data into training and test set (80:20). Find which model works better on the test set and use it to generate the daily consumption forecast one day at a time for the next 10 days. 10-days later, I performed a retro analysis as to which method came closest to the actuals and I have a results file for each customer's daily consumption for the 10 days. Using this I generate the following:
- Win % of all models (e.g. How many times did the manual forecast come closer to actual? 1:Yes,0:No) (Win % -Manual, Win%- ARIMA, Win%-TBATS)
- Overall Error (e.g Error-ARIMA, Error-Manual, etc.)
Here is my dilemma for making a recommendation:
Win % of min. error model on the test data is at 44%. ie of all the cases studied the manual forecast was closer to the actuals than the models But the overall error/deviation of the models as measured is lower than manual forecast.
Most of the models had a win % of 57% and, had a lower error/deviation than the manual forecast.
If I want to make a recommendation based on these results, which method works better the man or the machine?