2
$\begingroup$

Somewhat new to the forecasting area. I am trying to evaluate whether the statistical forecasts are better than manually generated forecasts in one of our used cases. I have 1000s of customers who consume our products daily. A broad question that comes to my mind is: How will you decide which methods works better on this problem of forecasting daily consumption? So, I conducted a man. vs. machine. exercise.

  1. Man = Manually generated forecasts
  2. Machine = ARIMA, Exp. Smoothing, TBATS etc.

I split the data into training and test set (80:20). Find which model works better on the test set and use it to generate the daily consumption forecast one day at a time for the next 10 days. 10-days later, I performed a retro analysis as to which method came closest to the actuals and I have a results file for each customer's daily consumption for the 10 days. Using this I generate the following:

  1. Win % of all models (e.g. How many times did the manual forecast come closer to actual? 1:Yes,0:No) (Win % -Manual, Win%- ARIMA, Win%-TBATS)
  2. Overall Error (e.g Error-ARIMA, Error-Manual, etc.)

Here is my dilemma for making a recommendation:

Win % of min. error model on the test data is at 44%. ie of all the cases studied the manual forecast was closer to the actuals than the models But the overall error/deviation of the models as measured is lower than manual forecast.

or

Most of the models had a win % of 57% and, had a lower error/deviation than the manual forecast.

If I want to make a recommendation based on these results, which method works better the man or the machine?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.