0
$\begingroup$

I forecasted/simulated a time series with a n step ahead forecast (n-ahead = 250) with 4 different time series models. Now I want to test, which of these models fits the best for the data.

All I found were methods, were we need the actual future values to compare (e.g RMSE, $R^2$ etc.). Are there techniques to test for a goodness of fit without these future values? Also I am interested in methods to compare the different forecasts.

$\endgroup$
1
$\begingroup$

In-sample fit is a very bad proxy for out-of-sample accuracy, and optimizing in-sample fit will tempt you to overfit.

The gold standard is to use a holdout sample: hold back the last observations of your historical data, fit your models to the remaining data, then forecast into the holdout sample and see which model performs best on it. This will not help you much if you need a long-range forecast, with a horizon longer than your history - but "standard" time series forecasting algorithm are not really made for such long-range forecasts, anyway.

If you compare models from the same model class, you can use information criteria such as AIC or BIC, which trade off in-sample fit against model complexity. So you can compare different ARIMA models, or different ETS models. However, you cannot compare an ARIMA model against an ETS model using information criteria.

I recommend Forecasting: Principles and Practice by Rob J Hyndman and George Athanasopoulos.

$\endgroup$
  • $\begingroup$ I am almost done with my analysis. My professor said he don't want me to cut my data in in-sample and out of sample. Also, my "forecast" isn't really one it is more like a simulation masked as a forecast (because I simulate unknown values which are supposed to lie in the future). The AIC and AICC I used to determine the best model order for each model. $\endgroup$ – user2968163 Nov 21 '17 at 8:11
  • 1
    $\begingroup$ Has your professor explained why he doesn't want you to use a holdout sample? $\endgroup$ – Stephan Kolassa Nov 21 '17 at 8:25
  • $\begingroup$ No. I suggested to him how I would proceed (doing 1 step ahead rolling or extending window forecasts, doing exactly this what you suggested and using RMSE, Mincer-Zarnowitz-regression, etc. to check for goodness of fit) but he wanted a I guess somewhat "theoretical" analysis. My explain would be, that he is not a stats or econometrics professor (he is a math prof) so the practical application is in the background. $\endgroup$ – user2968163 Nov 21 '17 at 8:35
  • 1
    $\begingroup$ It sounds like your challenge lies less with time series than with your professor. Perhaps Academia.SE is helpful. $\endgroup$ – Stephan Kolassa Nov 21 '17 at 8:38
  • $\begingroup$ Don't get me wrong he is one of the best professors I ever had (I also had and have some classes with him) and the supervision is great (always available, always takes time to answer my questions and assist me). But I am not a math student (i study finance with emphasis on financial mathematics) $\endgroup$ – user2968163 Nov 21 '17 at 8:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.