2
$\begingroup$

I'm trying to evaluate some software for forecast accuracy. It works by summing up all the orders from a number of locations for each month, then determines the best model out of a series of models based on the one the generates the minimum MSE. The it takes that model to forecast the demand for each location. For example, for Jan-Jun, Location A has demand (1,0,2,0,0,3) and Location B has demand (2,1,0,0,3,1). The aggregate would be A+B =(3,1,2,0,3,4). The software would then build models using ses, holt, MA, Croston's and Weighted Average. The one that produces the smallest MSE (in-sample) would be chosen to build the forecast for July. Then it would do the same thing again for August when it has an actual demand for July. It continues this way and may change the forecasting method at each month based on the minimum MSE. Therefore, it may generate forecasts for July-Dec using methods like, for example, (ses, ses, MA, Croston's ses, holt).

I currently have data from Jan 2016 to Dec 2017 (24 months) and I'm looking for advice regarding how to determine how well the tool determines a forecast. I thought about using tsCV, but that assume the same model will be applied each month in a rolling forecast, which isn't the case.

Improve this question
$\endgroup$
6
  • $\begingroup$ @SecretAgentMan: MAD/Mean is not a good idea, especially for intermittent demands. If you try to minimize this error measure, you may end up with "optimal" flat zero forecasts. See here for details and a few pointers to literature, and here for why this effect occurs. $\endgroup$
    – Stephan Kolassa
    Sep 14, 2018 at 15:45
  • $\begingroup$ @SecretAgentMan: the Smart-Willemain method is nice, but it cannot deal with dynamics in the time series, like trend, seasonality or causal factors. In addition, it is patented, which may be an IP problem for some practitioners. $\endgroup$
    – Stephan Kolassa
    Sep 14, 2018 at 15:46
  • $\begingroup$ @StephanKolassa, Thank you for the clarification. Since you're probably the SK from the forecasting book I have, I'll remove my comment until I find evidence to present with it. $\endgroup$
    – SecretAgentMan
    Sep 14, 2018 at 15:53
  • $\begingroup$ @StephanKolassa at the Foresight Practitioner conference, there were talks specifically encouraging using this metric for ID. Further, Willemain gives out R code for the Markov Bootstrap (without the jittering) but your point on the IP is well-taken. Thanks for the links. $\endgroup$
    – SecretAgentMan
    Sep 14, 2018 at 15:55
  • $\begingroup$ @SecretAgentMan: unfortunately, there is a lot of misinformation floating around in forecasting circles, including error metrics. Please take a look at the arguments I present in the linked threads, and if you find a hole in them, do let me know. In the meantime, you may want to run a little experiment: simulate a large number of iid Poisson variates $y_i$ for a parameter $\lambda<\ln 2$. Since they are iid, an optimal forecast will be a constant $\hat{y}$. Try different values for $\hat{y}$ and see which one minimizes MAD/Mean. $\endgroup$
    – Stephan Kolassa
    Sep 14, 2018 at 19:41

1 Answer 1

Reset to default
9
$\begingroup$

First off, don't use the in-sample accuracy to choose a model. This will invariably lead to overfitting. In-sample accuracy is not a good guide to out-of-sample prediction. Instead, use a holdout sample.

Regarding your main question: again, use a holdout sample to see how well your algorithm performs on truly new data.

Thus, if you are interested in $h$-month-ahead forecasts:

  1. Fit your models to the data except for the last $2h$ months.
  2. Forecast all of them out to a horizon of $h$ months. Note the forecast error of each model, using or whatever.
  3. Pick the model that performed best. Re-fit this model to the data except the last $h$ months. Forecast $h$ months ahead. Note the forecast error.

Do this for all your time series. Check how well this algorithm worked, and compare it to the performance of a few very simple benchmark methods, like always forecasting the historical mean, or the last observation. Or taking the average of all your candidate models' forecasts - averages of forecasts often outperform choosing the "best" method by some criterion.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Stephan Kolassa is perfectly right. I'd just like to add a purely business criteria. Which forecast will have the least negative impact on the business in case of error ? $\endgroup$
    – AlainD
    Sep 14, 2018 at 14:44
  • 1
    $\begingroup$ +1, " don't use the in-sample accuracy to choose a model" $\endgroup$
    – SecretAgentMan
    Sep 14, 2018 at 15:38
  • $\begingroup$ Stephan, if I understand you correctly, if I have a series for Jan-Dec like (1,0,0,2,5,2,0,1,2,1,2,0), then, if h=1, I should use the first 10 values to fit a series of models, i.e., ses, holt, MA, etc. Then forecast each model out to t=11 and determine the error using MAE, for example. Then pick the model with the smallest MAE and refit it to the 11 values to forecast out to value 12 and note the new MAE. Then do the same thing all over starting with the first 10 values using benchmark methods like Mean, Naive, etc. Do I have that correct? $\endgroup$
    – Angus
    Sep 14, 2018 at 18:44
  • $\begingroup$ @Angus: yes, that is exactly correct. $\endgroup$
    – Stephan Kolassa
    Sep 14, 2018 at 19:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.