0
$\begingroup$

Consider 3 models (model1, model2, model3) with the following set of prediction errors:

residuals_model1=c(-0.0422,-0.0198,-0.0167,-0.0350,-0.0084,-0.0387,0.0232,0.0239,-0.0104,-0.0174,0.0292,-0.0169,0.0456,-0.0068,0.0372,0.0557,0.0003,-0.0090)
residuals_model2=c(0.0109,-0.0129,-0.0058,0.0192,0.0055,0.0288,-0.0285,-0.0254,0.0013,0.0089,-0.0317,0.0319,-0.0262,0.0140,-0.0156,-0.0404,0.0190,0.0308)
residuals_model3=c(0.0189,0.0035,0.0025,0.0222,0.0004,0.0340,-0.0232,-0.0238,0.0145,0.0238,-0.0215,0.0243,-0.0379,0.0092,-0.0322,-0.0541,-0.0042,0.0111)

Computing root mean squared forecast errors (RMSFE):

rmsfe_model1=sqrt(mean(residuals_model1^2))
rmsfe_model2=sqrt(mean(residuals_model2^2))
rmsfe_model3=sqrt(mean(residuals_model3^2))

Next, create a matrix with the relative RMSFE of models 2 & 3 versus model 1 and the relevant p-values from the Diebold-Mariano test using the forecast package:

library(forecast)
forecast_results=matrix(nrow=2,ncol=2)
rownames(forecast_results)=c('model2 / model1','model3 / model1')
colnames(forecast_results)=c('relative RMSFE','p-value (DM-Test)')
forecast_results[1,1]=round(rmsfe_model2/rmsfe_model1,2)
forecast_results[2,1]=round(rmsfe_model3/rmsfe_model1,2)
forecast_results[1,2]=round(dm.test(residuals_model1,residuals_model2,alternative='two.sided',h=1,power=2)$p.value,2)
forecast_results[2,2]=round(dm.test(residuals_model1,residuals_model3,alternative='two.sided',h=1,power=2)$p.value,2)

Print results:

print(forecast_results)

                relative RMSFE p-value (DM-Test)
model2 / model1           0.79              0.10
model3 / model1           0.85              0.04

It can be seen that model 2 performs better in terms of relative RMSFE compared to model 3. However, the p-value from the DM-test shows a higher level of statistical significance for model 3 compared to model 2.

Suppose you had to present this table of results and somebody would claim that a higher relative RMSFE for model 3 is inconsistent with a lower p-value for model 3. How would you explain this result?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ The residuals plots show a break in the last period. If you exclude that, the p-values are almost the same, as model2 and model3 are not different according to dm.test. Why your reference model is negative correlated (almost perfect in one case) with the other models? $\endgroup$
    – Robert
    Commented Aug 15, 2016 at 12:45
  • $\begingroup$ Thank you for pointing out the issue with the break in the last period! And good point regarding the negative correlation, I have to investigate this as it looks a bit suspicious. $\endgroup$
    – kanimbla
    Commented Aug 15, 2016 at 13:40
  • $\begingroup$ Regarding the correlation, most probably the sign of the residual vector from model 1 or the sign of the residual vectors from models 2 & 3 should be reversed (note that this would not change the above results). Model 1 refers to an autoregressive model and models 2 & 3 refer to an autoregressive distributed lag model. Side note: ARDL models are estimated via Bayesian Model Averaging so the AR model is not a fully nested version of the former. $\endgroup$
    – kanimbla
    Commented Aug 15, 2016 at 13:59

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.