I'm to write a short report on Time Series forecast comparison. I'm a beginner in the field.

I want to investigate how one chooses which model is better than the other based on the forecast results.

So far I've stumbled upon nmse, rmse, mse, and other variations of the standard error.In random papers I googled I noticed that the author presents 2 or 3 different models along with their errors and chooses the model with the smallest one.

Are these the only metrics I need to look at before I make my decision?

Are there any statistical tests that will help me better decide?

Any answer and/or pointer to a paper or a book is greatly appreciated!

  • $\begingroup$ how much data do you have ? and what is the frequency of your data - daily, weekly, monthly, quarterly or yearly ? $\endgroup$ – forecaster Jan 11 '15 at 2:56
  • $\begingroup$ You want an absolute measure (like RMSE, MAPE), a bias measure (like MPE) and, if you have a time series, a measure of residual autocorrelation. As to books, check out section 2.5 of Hyndman and Athana­sopou­los's book, conveniently online. otexts.org/fpp/2/5 $\endgroup$ – zbicyclist Jan 11 '15 at 3:57
  • $\begingroup$ A good forecasting method should make random mistakes but not systematic ones. The forecast errors should be unpredictable. As one of the simplest illustrations, the method should not overpredict or underpredict systematically. $\endgroup$ – Richard Hardy Jan 11 '15 at 12:32
  • $\begingroup$ @forecaster I don't work on specific data. I have to make a report on how one chooses which forecasting model is better in general, so I'm more interested in methodologies. Are you suggesting different frequencies require different methodologies? $\endgroup$ – themistoklik Jan 11 '15 at 17:31
  • $\begingroup$ Yes, specifically measures like Mean absolute scaled error are different for seasonal data vs non seasonal data. In addition length of data is more important because that would dictate if you can hold out the data for testing. $\endgroup$ – forecaster Jan 11 '15 at 18:53

There are two surveys that I can think of that cover this topic systematically.
1. The best known is the one by Hyndman and Koehler (2006) (see supporting materials here), which motivates MASE as the preferred criterion for forecast evaluation.
2. There is also a comprehensive survey chapter by Ken West in the first volume of the Handbook of Forecasting, which was updated by Clark & McCracken in volume 2B of the Handbook of Forecasting.

The latter are far more theoretical but the Hyndman and Koehler paper is very accessible.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.