Somewhat new to the forecasting area. I am trying to evaluate whether the statistical forecasts are better than manually generated forecasts in one of our used cases. I have 1000s of customers who consume our products daily. A broad question that comes to my mind is: How will you decide which methods works better on this problem of forecasting daily consumption? So, I conducted a man. vs. machine. exercise.

  1. Man = Manually generated forecasts
  2. Machine = ARIMA, Exp. Smoothing, TBATS etc.

I split the data into training and test set (80:20). Find which model works better on the test set and use it to generate the daily consumption forecast one day at a time for the next 10 days. 10-days later, I performed a retro analysis as to which method came closest to the actuals and I have a results file for each customer's daily consumption for the 10 days. Using this I generate the following:

  1. Win % of all models (e.g. How many times did the manual forecast come closer to actual? 1:Yes,0:No) (Win % -Manual, Win%- ARIMA, Win%-TBATS)
  2. Overall Error (e.g Error-ARIMA, Error-Manual, etc.)

Here is my dilemma for making a recommendation:

Win % of min. error model on the test data is at 44%. ie of all the cases studied the manual forecast was closer to the actuals than the models But the overall error/deviation of the models as measured is lower than manual forecast.


Most of the models had a win % of 57% and, had a lower error/deviation than the manual forecast.

If I want to make a recommendation based on these results, which method works better the man or the machine?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.