9
$\begingroup$

Introduction

In forecasts combination one of the popular solutions is based on the application of some information criterion. Taking for example Akaike criterion $AIC_j$ estimated for the model $j$, one could compute the differences of $AIC_j$ from $AIC^* = \min_j{AIC_j}$ and then $RP_j = e^{(AIC^*-AIC_j)/2}$ could be interpreted as the relative probability of model $j$ to be the true one. The weights then are defined as

$$w_j = \frac{RP_j}{\sum_j RP_j}$$

Problem

A difficulty that I try to overcome is that the models are estimated on the differently transformed response (endogenous) variables. For example, some models are based on annual growth rates, another - on quarter-to-quarter growth rates. Thus the extracted $AIC_j$ values are not directly comparable.

Tried solution

Since all that matters is the difference of $AIC$s one could take the base model's $AIC$ (for example I tried to extract lm(y~-1) the model without any parameters) that is invariant to the response variable transformations and then compare the differences between the $j$th model and the base model $AIC$. Here however it seems the weak point remains - the difference is affected by the transformation of the response variable.

Concluding remarks

Note, the option like "estimate all the models on the same response variables" is possible, but very time consuming. I would like to search for the quick "cure" before going to the painful decision if there is no other way to resolve the problem.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I think one of the most reliable methods for comparing models is to cross-validate out-of-sample error (e.g. MAE). You will need to un-transform the exogenous variable for each model to directly compare apples to apples.

Improve this answer
$\endgroup$
6
  • $\begingroup$ An alternative way that I have left for even more time consuming approach is to use the jack-knifed errors to estimate the weights similar to Bates and Granger (1969) and the related works like Clements and Harvey Forecasts combinations and encompassing (2007). The weak point of forecast errors based approach is that it is on average inferior to information (model) based approaches. Since the Bayesian averaging is tricky, I tried to apply a more simple method that could be thought of being BMA with informative priors. $\endgroup$
    – Dmitrij Celov
    Commented Dec 15, 2011 at 13:49
  • $\begingroup$ Note, that I neither want to compare and select the best model, nor am searching for the best forecasts combination method. I simply have problems comparing the AICs from the models based on differently transformed response variables. $\endgroup$
    – Dmitrij Celov
    Commented Dec 15, 2011 at 13:51
  • 1
    $\begingroup$ @Dmitrij Celov: Then why are you comparing AICs? Keep in mind that AIC is asymptotically equivalent to leave-one-out cross-validation, so I suspect comparisons of either metric would be similar. stats.stackexchange.com/a/587/2817 $\endgroup$
    – Zach
    Commented Dec 15, 2011 at 13:56
  • $\begingroup$ @DmitrijCelov: "The weak point of forecast errors based approach is that it is on average inferior to information (model) based approaches." Inferior in what regard? Do you have some citations or explanation for this? Intuition tells me this statement is wrong, but intuition is often wrong... $\endgroup$
    – Zach
    Commented Dec 15, 2011 at 13:58
  • $\begingroup$ I probably made a quick conclusion after the remark in G.Kapitanious et al working paper Forecast combinations and the Bank of England's suite of statistical forecasting methods where on p. 23 it is written that "...combining forecasts will not in general deliver the optimal forecast, while combining information will". Asymptotic equivalence is not what I would like to have in small samples of macroeconomic data, but simple methods may outperform more complex. Simply cross-validation is the second best solution, jack-knives are produced within one week, AICs in an hour. (We may go to chat) $\endgroup$
    – Dmitrij Celov
    Commented Dec 15, 2011 at 14:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.