Suppose I have two in-sample forecasts from two different non-nested models. I want to check which one produces the best forecasts. A common way is Diebold-Mariano, GiacominiWhite, ENC-T test. However, these tests were designed for out-of-sample forecasts. Would they also work in-sample?

So, in my case, I have a time-invariant model and the same model with time-varying coefficients. I want to check whether it is important to account for time-variation in coefficients. Does AIC still work then? I thought of comparing the R2 in both models, 1 minus tge ratio of the sum of the squared residuals in both models. The issue is where to draw the line. Suppose I have unconditional model R2 of 5% and conditional one of 6 or 7% (I am dealing with asset pricing predictive regressions so the R2s are often even smaller than this). How to decide that this improvement of 1 ,2 3% in the R2 is economically significant based on R2, AIC and so on based on in-sample methods?

  • $\begingroup$ Why don't you do 1-day ahead predictions by re-estimating the model daily vs the fixed parameter model predictions. Then you can do Diebold-Mariano. Make sure too keep a constant window length for your daily re-estimations. $\endgroup$
    – stollenm
    Apr 19 '18 at 6:09
  • $\begingroup$ Daniel, see my answers here and here. They were written for different questions but might be somewhat relevant to this one, too. $\endgroup$ Apr 20 '18 at 16:21

In sample, you would look at goodness-of-fit measures, such as Bayes Information Criterion (BIC), Akaike Information Criterion (AIC) or R^2.

Generally speaking, the more parameters your model has, the better you can fit the data. For this reason there it is difficult to define a "fair" way to compare in sample fit. BIC and AIC have penalty terms for the number of parameters but theses are chosen somewhat heuristically.

  • $\begingroup$ Thanks @stollenm. I think I should have explained the issue better I just added a small paragraph above. $\endgroup$ Apr 18 '18 at 12:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.