Okay, I have a couple books on time series forecasting, but perhaps I need to read a couple more.

Here's my question.

You want to be able to validate a forecasting model. So you split the data into "training" data and "test" data --- to build a model off the "training" data and then use the test data as an out-of-sample trial run to analyze how well the model predicts future unknown data.

So, let's say a triple exponential smoothing model (with dampening) turns out to be the best fit model on the training data. Alpha, beta, gamma (and phi) parameters are optimized, initial levels, trend, and seasonality are determined. You test it on the test data, and the MAPE, ME, RMSE, MAE seem fine.

Now, here's the question. Is this model -- built on training data (80% whole data) -- parameters and everything -- is THIS the model you're going to keep for future forecasting (you will use the training model parameters with 100% of the data inserted).

Or would you think --- hmmm.. the parameter optimization and initial levels worked out well for the training data ... I'm going to re-run the optimization for 100% of the data now. See what I'm saying? Are you validating the training model, or the process by which you ascertained the training model?

Also, I tried an experiment whereas I applied an 80% training data based model to forecast the last 10% of the data .... vs. a 90% training data model to forecast the last 10% of the data (both had 90% of the data as historical inputs).

The RMSE and MAPE of the 80%-training model were 10-20% higher than the 90% model. So it would seem updating the model as new data comes in --- at least only based on this one observation --- might be beneficial to forecast accuracy.

However, the autocorrelation of residuals seems to have a major difference. The 80%-training forecast has an Auto-correlation of -0.017 among residuals in the forecast, whereas the 90% training forecast has an AC of -0.278 among residuals. I know the residuals are supposed to indicate the goodness-of-fit of a model, but when it comes to dollars and cents, aren't RMSE and MAPE really the gold standard (RMSE if errors cause exponential catastrophe and MAPE if errors scale linearly with consequences).

Let me know if this makes sense.

Also, as a brief aside, when using ets() function in R, it doesn't seem to always choose the best model based on RMSE/ MAPE/ or even AIC. I'm not sure why this is.

  • $\begingroup$ I believe RMSE is about quadratic loss, not exponential. Also, I suggest you make your question stand out visually (use double asterisks from both sides for boldface font). To me it seems your main question is, in your own words, Are you validating the training model, or the process by which you ascertained the training model? I assume you understand how to do both but you are wondering which one to go for. The simple way would be to validate the model (as opposed to the procedure) and later allow the parameters to be reestimated whenever new data arrives. Do I get you question alright? $\endgroup$ – Richard Hardy Nov 24 '14 at 20:48
  • 1
    $\begingroup$ Perhaps there are three alternatives, two of which you had in mind. (1) Given a model class (e.g. all ARMA models), select model order (e.g. ARMA(2,1)). For forecasting, use the same model order but reestimate parameters based on all available data. (2) Given a model class AND model order, see how well it does in the test sample (but no model tuning except for parameter estimation). Reestimate parameters based on all available data before forecasting. (3) Given a model class AND model order, see how well it does in the test sample. Do NOT reestimate parameters before forecasting. $\endgroup$ – Richard Hardy Nov 24 '14 at 21:00
  • $\begingroup$ In the last example, I think (3) does not make much sense while (1) could often be the reasonable choice. $\endgroup$ – Richard Hardy Nov 24 '14 at 21:01
  • $\begingroup$ Thanks. Yes the 1st suggestion is what I was thinking about. I was hesitant about recalculating the parameters, because I've heard a few times that 'constantly optimizing' the parameters (specifically alpha, beta, gamma for exponential smoothing models) --- ultimately may produce worse forecast accuracy then optimizing once. Of course I'm not an expert as to why. It seems like at minimum, tuning them every 3-6 months would do much more help than harm. I was wondering if I re-estimated the parameters, that would make the new model untested/ potentially overfitted, but looking back, probably not $\endgroup$ – John Babson Nov 25 '14 at 17:08
  • 1
    $\begingroup$ It will not overfit if it is carefully designed. On the other hand, not using all the available information when building a model seems suboptimal in principle. When a new piece of data arrives, why would one limit its contribution to parameter estimation but neglect it when estimating model order? A priori theoretical restrictions could be a reason, but otherwise it makes little sense to do so. $\endgroup$ – Richard Hardy Dec 1 '14 at 18:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.