0
$\begingroup$

I am testing forecasts against against realized values with a number of observations of 4000.

When I calculate the mean MAE, its relatively low. Around 10% deviation to the original variable.

However, when I regress the forecasts against the realized values I get an r-squared value of 0.06...

Is there any rationale behind this?

How this can be?

Extrem outliners? Or extrem bias? Or something else?

The MAE is calculated as $mean(abs(y_{real,t+1} - y_{predict,t+1}))$. The t+1 indicating that I only forecast and compare one-step-ahead forecasts.

and the regression is simply calculated by MATLAB with the function regstats,which regress $y_{real}$ over $y_{predict}$.

I do the regression during the mincer-zarnowitz test. It completely fails.

This is the plot: enter image description here

$\endgroup$
  • $\begingroup$ You need to provide us with the expressions for a) the calculation of MAE b) the regression equation of forecasts on realized values $\endgroup$ – Alecos Papadopoulos Jan 7 '17 at 18:51
  • $\begingroup$ Also, are you examining one-step-ahead forecasts only? $\endgroup$ – Alecos Papadopoulos Jan 7 '17 at 18:55
  • $\begingroup$ yes. Only one-step-ahead forecasts $\endgroup$ – Alexander De Beur Jan 7 '17 at 19:37
  • $\begingroup$ Could you show a plot of the data? Visualizing your problem can be really helpfull to see what is going on. $\endgroup$ – Pieter Jan 7 '17 at 19:44
  • $\begingroup$ Your data points are not really independent... If you want to assess your model fairly you should probably sample the data to get rid of the temporal dependency. $\endgroup$ – Pieter Jan 7 '17 at 20:03
0
$\begingroup$

for me it seems that there is a time gap between the realized and forecasted values.

Maybe you are trying to forecast new values for some future horizon based on the old values. When the future is not behaving as the past you will have high deviations at the beginning and the end of your testing period.

In this case the Regression indicate that there is no relation in your data although it is clear from the plot that there is a strong relation... just image the points at the right lower corner weren't there.... check these points.

$\endgroup$
  • $\begingroup$ It seems you are right. I am using past data (past one year) to predict the future one year... I guess I have to life with the fact, that the forecasts are biased according to Mincer zarnowitz... I cannot change the facts. For me, this methods seems also a bit stupid to be honest, because some outliners, or such "time lag" effects leading to such results, although the MAE is quite ok... like 10%. Additionally I will now use Diebold Mariano to at least compare the forecasts. However, half of my work crushed $\endgroup$ – Alexander De Beur Jan 8 '17 at 14:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.