1
$\begingroup$

I constructed a VAR model of order 4 where some of the variables are statistically insignificant. The model is based right in terms of diagnostics (no autocorrelation of residuals, normal distribution, homoskedasticity). I then removed the statistically insignificant variables and computed forecasts.

However, when I evaluate the quality of forecasts using root mean square error (RMSE), I obtain very high values; for example, the response variable has values around 100 while RMSE comes in at around 50.

Is it possible?
How to solve such a conflict: I have good model and such poor forecasts?
Did I make mistake somewhere?

$\endgroup$
2
  • 1
    $\begingroup$ By "good model" how did you ascertain that? Also, if you only want to use significant variables, you should rebuild your model by applying restrictions. $\endgroup$
    – Arun Jose
    Commented Jul 20, 2016 at 10:27
  • $\begingroup$ I was going through my old answers and noticed this one was not accepted. Do you perhaps need further clarification? $\endgroup$ Commented Feb 19, 2017 at 9:18

1 Answer 1

2
$\begingroup$

Some variables are intrinsically harder to predict than others. Even if you build a model following all the best-practice guidelines, you may not always accurately predict a process just because it contains a large component of randomness. RMSE values by themselves do not tell the whole story.

However, you may sensibly evaluate the forecasting performance by taking into account

  1. subject-matter knowledge (e.g. stock returns are really hard to predict);
  2. related studies (can you match or beat their forecast accuracy?);
  3. simple benchmarks (does your model of choice do at least as well as a simple benchmark model?).
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.