0
$\begingroup$

I'm working on a sales forecasting package which should be easy to use for the end user. Given a time series with historical sales data I would like to automatically select one of the three forecasts: Auto.Arima, ETS and STLF. The idea is to split historical data into 80% train set and 20% test (holdout) set. Then run Auto.Arima, ETS, STLF and choose the one that has best MAPE on the test set.

Now comes the part that is not entirely clear to me. Once I figured out that e.g. ETS gives me the best result should I now

  1. Retrain ETS on the entire set of historical data and generate forecast using this new model? My reservation here is that after I run ETS again it may even change the class of the algorithm as well as the fit parameters which will render the MAPE I got on the test set irrelevant.
  2. Just generate the forecast using the model that was trained on the 80% train set? My problem with this approach is that we are ignoring the last 20% of data which is probably the most important information for the forecast.
  3. The third idea is to use the same model fit parameters that we got after training the model on the 80% train set. But then use the entire set of data for forecasting. This seems like a reasonable approach but I cannot figure out how to do it for ETS and STL (For Arima we can do it by supplying the original fit as the model parameter of the arima function)

Could you please let me know what is the right way to approach this problem?

|cite|improve this question
$\endgroup$
  • $\begingroup$ Question is ambiguous: do you seek code in R or do you want advice on strategy? The latter would be on-topic here, except that "the right way" is difficult to determine either in general or for your datasets. $\endgroup$ – Nick Cox Dec 22 '15 at 10:19
  • $\begingroup$ Hi Nick. First of all I need an advice on strategy. But in case the recommended approach is number 3 then I will need some help with implementing it in R because, as I said, I understand how to implement it for Arima (stats.stackexchange.com/questions/55937/…) but not for ETS or STL. $\endgroup$ – Ivan Lebedev Dec 22 '15 at 11:35
  • $\begingroup$ Software-specific advice is off-topic here. but the statistical question remains. $\endgroup$ – Nick Cox Dec 22 '15 at 11:36
1
$\begingroup$

By using 80% and retaining 20% the dog is being wagged by the tail AND even more importantly you are using a sample of 1 origin rather than multiple origins to determine "best". A good procedure is to determine how long (periods ahead) you want to typically forecast (could be 1 period or k periods) say 3 periods for example . Now take your data set say N historical observations and build a model/assess forecasts say from 6 origins i.e. NOB-3, NOB-6 ,... NOB-18 . Construct the 6 models and project each three periods hence. Compute Mapes based upon the 6 origins. In terms of model selection make sure that you consider hybrid models using both ARIMA (memory) and deterministic structure like level shifts, time trend, seasonal pulses and pulses. These models are call Robust Transfer Functions or Dynamic Regression with Intervention Detection.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ If I understood correctly you are suggesting to use the "rolling forecasting origin" method described here (otexts.org/fpp/2/5). What is still unclear to me is if I train, say, ARIMA first on NOB-3, then on NOB-6, e.t.c. then I will have 6 different models for each of the methods being cross-validated (ARIMA, ETS, STL). How should I generate the forecast once I've identified the best method. Should I now retrain the winning method on the entire set of historical data and generate the forecast using the resulting model? $\endgroup$ – Ivan Lebedev Dec 23 '15 at 7:51
  • $\begingroup$ "Training on NOB-3" generates 1 model based upon NOB-3 observations and is then only used to predict the next 3 ,,, thus 6 models are developed and all are used to predict 3 subsequent values. What is being measured here is the general/overall value of a particular method from 6 origins. $\endgroup$ – IrishStat Dec 23 '15 at 10:35
  • $\begingroup$ I understand that. But when I've identified the best method (e.g. ETS) I want to use it to predict the 3 values in the future. Should I now train ETS on the entire set of historical data and use this new model fit to predict the 3 future values? $\endgroup$ – Ivan Lebedev Dec 23 '15 at 12:15
  • $\begingroup$ No...just compute the average ( over all origins) to get a measure of each method. $\endgroup$ – IrishStat Dec 23 '15 at 13:19
  • $\begingroup$ @IvanLebedev, I would say, yes, retrain on the entire sample and use that to predict out of sample -- unless you suspect the data generating process to be changing over time so that the early observations are little representative of what you are going to forecast. (It seems to me that the last comment before mine addresses something else than actual out-of-sample forecasting.) $\endgroup$ – Richard Hardy Dec 23 '15 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.