0
$\begingroup$

I am testing forecasts against against realized values with a number of observations of 4000.

When I calculate the mean MAE, its relatively low. Around 10% deviation to the original variable.

However, when I regress the forecasts against the realized values I get an r-squared value of 0.06...

Is there any rationale behind this?

How this can be?

Extrem outliners? Or extrem bias? Or something else?

The MAE is calculated as $mean(abs(y_{real,t+1} - y_{predict,t+1}))$. The t+1 indicating that I only forecast and compare one-step-ahead forecasts.

and the regression is simply calculated by MATLAB with the function regstats,which regress $y_{real}$ over $y_{predict}$.

I do the regression during the mincer-zarnowitz test. It completely fails.

This is the plot: enter image description here

$\endgroup$
10
  • $\begingroup$ You need to provide us with the expressions for a) the calculation of MAE b) the regression equation of forecasts on realized values $\endgroup$ Commented Jan 7, 2017 at 18:51
  • $\begingroup$ Also, are you examining one-step-ahead forecasts only? $\endgroup$ Commented Jan 7, 2017 at 18:55
  • $\begingroup$ yes. Only one-step-ahead forecasts $\endgroup$ Commented Jan 7, 2017 at 19:37
  • $\begingroup$ Could you show a plot of the data? Visualizing your problem can be really helpfull to see what is going on. $\endgroup$
    – Pieter
    Commented Jan 7, 2017 at 19:44
  • $\begingroup$ Your data points are not really independent... If you want to assess your model fairly you should probably sample the data to get rid of the temporal dependency. $\endgroup$
    – Pieter
    Commented Jan 7, 2017 at 20:03

2 Answers 2

2
$\begingroup$

When I calculate the mean MAE, its relatively low. Around 10% deviation to the original variable.

The R-squared does not determine error relative to 'the size of the original variable', but instead relative to 'the size of the variance of the original variable'.

Your predictions are between 0.6 and 1 and may have only a small error up to $\pm 0.2$ which is small relative to the size of the variable which is around 0.7.

But the deviations of your predictions relative to the mean are not much the same as the deviations of the true variable relative to the mean. When the true value is high around 0.8, then your predictions can be just as well low around 0.6 as high around 1.0. When the true value is low around 0.6 then your predictions can be just as well low around 0.6 as high around 1.0.

Your predictions are around the same level as the true variable, but they do not capture the variations in the variable.

$\endgroup$
0
$\begingroup$

for me it seems that there is a time gap between the realized and forecasted values.

Maybe you are trying to forecast new values for some future horizon based on the old values. When the future is not behaving as the past you will have high deviations at the beginning and the end of your testing period.

In this case the Regression indicate that there is no relation in your data although it is clear from the plot that there is a strong relation... just image the points at the right lower corner weren't there.... check these points.

$\endgroup$
1
  • $\begingroup$ It seems you are right. I am using past data (past one year) to predict the future one year... I guess I have to life with the fact, that the forecasts are biased according to Mincer zarnowitz... I cannot change the facts. For me, this methods seems also a bit stupid to be honest, because some outliners, or such "time lag" effects leading to such results, although the MAE is quite ok... like 10%. Additionally I will now use Diebold Mariano to at least compare the forecasts. However, half of my work crushed $\endgroup$ Commented Jan 8, 2017 at 14:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.