- I wonder why general-to-specific (GETS) subset selection and particularly the Autometrics algorithm are performing well in macroeconomic modelling/forecasting.
How does Autometrics work?
- Doornik "Autometrics" (2009) offers a full description (see p. 7-8 for the algorithm scheme).
The algorithm is based on stepwise subset selection with qualifications and includes diagnostic testing.
A few main features are listed below.
- The general model is formulated as a linear model with
(i) main variables,
(ii) generous lags thereof, and in later versions also
(iii) impulses (for impulse indication saturation (IIS) or step indication saturation (SIS) techniques - to account for possible structural breaks) and
(iv) nonlinear transformations.
- The algorithm does stepwise selection starting from a general model and going down several paths.
- Variables may not only be removed but also added at each step (except the initial step where all the variables are included).
- Significance testing is used for variable removal/inclusion.
- When the search terminates (based on significance of variables), diagnostic testing is carried out. Poor models are removed from further consideration.
How well does Autometrics perform?
- Good performance of Autometrics is apparent from works of Sir David F. Hendry, Jurgen A. Doornik et al. (in the links, see works on model selection, Autometrics and related).
Of course, independent authors' experience would be more credible than the experience of the "fathers" of Autometrics.
- Kock & Teräsvirta "Forecasting performances of three automated modelling techniques during the economic crisis 2007–2009" (2014) find:
[Autometrics] works well when the model is a reasonable approximation to reality, but less well when it is not. In direct forecasting, one is facing the latter situation... It appears that Autometrics may not be an appropriate tool for building models for direct multiperiod forecasting. However, it can be an excellent choice when the data-generating process is approximated well by a subset of variables in the data set of the researcher.
Epprecht et al. "Comparing variable selection techniques for linear regression: LASSO and Autometrics" (2013) find that Autometrics and LASSO outperform each other in different respects and different settings.
The typical setting in Hendry's own performance testing of Autometrics is when the correct model is a subset of the general model. Autometrics is shown to select a model that is either the true one or very close to it.
Meanwhile, under tapering effects or considerable misspecification of the general model Autometrics may perform less well, e.g. as in Kock & Teräsvirta (2014) above.
Why is good performance surprising?
- With no shrinkage and having to select among a large number of variables (that is the setting in many of Hendry's and Doornik's works), I would anticipate GETS to fail. (E.g. Frank Harrell has condemned subset selection in several posts, e.g. 1 and 2, although the posts were not specific to macroeconomic time series.)
- (Question) Could the good performance of GETS and Autometrics be specific to the applications (mostly macroeconomic time series modelling),
or is Autometrics just a really good method in a broad sense?
"LASSO/LARS vs general to specific (GETS) method" is a related question.