# statistical significance of time time-series forecasts

I've created three different time-series forecasts with three different models. One is a simple average, the second is an ARMA model and the third is a SETAR model. The models were applied on a moving window of 20 days. Since I use intraday data, the window uses 20 days to forecast the next day and so on.

After getting the results from the three models, I now want to test which of my predictions is the "statistically" best. I compared each forecasting result with the "real" observations.Hence, my hypothesis looks as follows:

H_0: The forecast and the observations are equally accurate on average
H_1: The forecast and the observations are not equally accurate on average


Then, I run an OLS regression with the observations of the real data as the exogenous variable and the forecasts of the models as the endogenous variable. I do this three times, for each model once against the observations. Now I'm left with three different results from the OLS regression. To decide which of the models performed (statistically significant) best, I compare the R-squares, the standard error and the t-tests.

Is my hypothesis and my testing approach appropriate to test which model performed best, or am I doing something completely wrong?

• Why not just compare, for example, the mean squared errors of the actuals - the forecasts? – jbowman Jul 13 '18 at 1:23
• I already did that, as well as some other measures error measores like MAPE. But I'm not sure if that is enough to proof that my forecast is statistically significant. – Krasnars Jul 13 '18 at 4:46