1
$\begingroup$

I've created three different time-series forecasts with three different models. One is a simple average, the second is an ARMA model and the third is a SETAR model. The models were applied on a moving window of 20 days. Since I use intraday data, the window uses 20 days to forecast the next day and so on.

After getting the results from the three models, I now want to test which of my predictions is the "statistically" best. I compared each forecasting result with the "real" observations.Hence, my hypothesis looks as follows:

H_0: The forecast and the observations are equally accurate on average 
H_1: The forecast and the observations are not equally accurate on average

Then, I run an OLS regression with the observations of the real data as the exogenous variable and the forecasts of the models as the endogenous variable. I do this three times, for each model once against the observations. Now I'm left with three different results from the OLS regression. To decide which of the models performed (statistically significant) best, I compare the R-squares, the standard error and the t-tests.

Is my hypothesis and my testing approach appropriate to test which model performed best, or am I doing something completely wrong?

$\endgroup$
  • $\begingroup$ Why not just compare, for example, the mean squared errors of the actuals - the forecasts? $\endgroup$ – jbowman Jul 13 '18 at 1:23
  • $\begingroup$ I already did that, as well as some other measures error measores like MAPE. But I'm not sure if that is enough to proof that my forecast is statistically significant. $\endgroup$ – Krasnars Jul 13 '18 at 4:46
1
$\begingroup$

The standard approach to test whether one forecast yields statistically significantly better point forecasts than another one is the test. You may want to look through the tag, and the tag wiki contains pointers to literature.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.