Okay, I have a couple books on time series forecasting, but perhaps I need to read a couple more.
Here's my question.
You want to be able to validate a forecasting model. So you split the data into "training" data and "test" data --- to build a model off the "training" data and then use the test data as an out-of-sample trial run to analyze how well the model predicts future unknown data.
So, let's say a triple exponential smoothing model (with dampening) turns out to be the best fit model on the training data. Alpha, beta, gamma (and phi) parameters are optimized, initial levels, trend, and seasonality are determined. You test it on the test data, and the MAPE, ME, RMSE, MAE seem fine.
Now, here's the question. Is this model -- built on training data (80% whole data) -- parameters and everything -- is THIS the model you're going to keep for future forecasting (you will use the training model parameters with 100% of the data inserted).
Or would you think --- hmmm.. the parameter optimization and initial levels worked out well for the training data ... I'm going to re-run the optimization for 100% of the data now. See what I'm saying? Are you validating the training model, or the process by which you ascertained the training model?
Also, I tried an experiment whereas I applied an 80% training data based model to forecast the last 10% of the data .... vs. a 90% training data model to forecast the last 10% of the data (both had 90% of the data as historical inputs).
The RMSE and MAPE of the 80%-training model were 10-20% higher than the 90% model. So it would seem updating the model as new data comes in --- at least only based on this one observation --- might be beneficial to forecast accuracy.
However, the autocorrelation of residuals seems to have a major difference. The 80%-training forecast has an Auto-correlation of -0.017 among residuals in the forecast, whereas the 90% training forecast has an AC of -0.278 among residuals. I know the residuals are supposed to indicate the goodness-of-fit of a model, but when it comes to dollars and cents, aren't RMSE and MAPE really the gold standard (RMSE if errors cause exponential catastrophe and MAPE if errors scale linearly with consequences).
Let me know if this makes sense.
Also, as a brief aside, when using ets() function in R, it doesn't seem to always choose the best model based on RMSE/ MAPE/ or even AIC. I'm not sure why this is.