I'm working on a sales forecasting package which should be easy to use for the end user. Given a time series with historical sales data I would like to automatically select one of the three forecasts: Auto.Arima, ETS and STLF. The idea is to split historical data into 80% train set and 20% test (holdout) set. Then run Auto.Arima, ETS, STLF and choose the one that has best MAPE on the test set.
Now comes the part that is not entirely clear to me. Once I figured out that e.g. ETS gives me the best result should I now
- Retrain ETS on the entire set of historical data and generate forecast using this new model? My reservation here is that after I run ETS again it may even change the class of the algorithm as well as the fit parameters which will render the MAPE I got on the test set irrelevant.
- Just generate the forecast using the model that was trained on the 80% train set? My problem with this approach is that we are ignoring the last 20% of data which is probably the most important information for the forecast.
- The third idea is to use the same model fit parameters that we got after training the model on the 80% train set. But then use the entire set of data for forecasting. This seems like a reasonable approach but I cannot figure out how to do it for ETS and STL (For Arima we can do it by supplying the original fit as the model parameter of the arima function)
Could you please let me know what is the right way to approach this problem?