General-to-specific subset selection ("Autometrics") performing well in macroeconomics

Question

• I wonder why general-to-specific (GETS) subset selection and particularly the Autometrics algorithm are performing well in macroeconomic modelling/forecasting.

How does Autometrics work?

1. Doornik "Autometrics" (2009) offers a full description (see p. 7-8 for the algorithm scheme).
   The algorithm is based on stepwise subset selection with qualifications and includes diagnostic testing.
   A few main features are listed below.
2. The general model is formulated as a linear model with
   (i) main variables,
   (ii) generous lags thereof, and in later versions also
   (iii) impulses (for impulse indication saturation (IIS) or step indication saturation (SIS) techniques - to account for possible structural breaks) and
   (iv) nonlinear transformations.
3. The algorithm does stepwise selection starting from a general model and going down several paths.
4. Variables may not only be removed but also added at each step (except the initial step where all the variables are included).
5. Significance testing is used for variable removal/inclusion.
6. When the search terminates (based on significance of variables), diagnostic testing is carried out. Poor models are removed from further consideration.

How well does Autometrics perform?

1. Good performance of Autometrics is apparent from works of Sir David F. Hendry, Jurgen A. Doornik et al. (in the links, see works on model selection, Autometrics and related).
   Of course, independent authors' experience would be more credible than the experience of the "fathers" of Autometrics.
2. Kock & Teräsvirta "Forecasting performances of three automated modelling techniques during the economic crisis 2007–2009" (2014) find:

[Autometrics] works well when the model is a reasonable approximation to reality, but less well when it is not. In direct forecasting, one is facing the latter situation... It appears that Autometrics may not be an appropriate tool for building models for direct multiperiod forecasting. However, it can be an excellent choice when the data-generating process is approximated well by a subset of variables in the data set of the researcher.

3. Epprecht et al. "Comparing variable selection techniques for linear regression: LASSO and Autometrics" (2013) find that Autometrics and LASSO outperform each other in different respects and different settings.

4. The typical setting in Hendry's own performance testing of Autometrics is when the correct model is a subset of the general model. Autometrics is shown to select a model that is either the true one or very close to it.
   Meanwhile, under tapering effects or considerable misspecification of the general model Autometrics may perform less well, e.g. as in Kock & Teräsvirta (2014) above.

Why is good performance surprising?

1. With no shrinkage and having to select among a large number of variables (that is the setting in many of Hendry's and Doornik's works), I would anticipate GETS to fail. (E.g. Frank Harrell has condemned subset selection in several posts, e.g. 1 and 2, although the posts were not specific to macroeconomic time series.)
2. (Question) Could the good performance of GETS and Autometrics be specific to the applications (mostly macroeconomic time series modelling),
   or is Autometrics just a really good method in a broad sense?

"LASSO/LARS vs general to specific (GETS) method" is a related question.

Answer 1

Frank Harrell does not rule out intelligent use of backward elimination. He includes as a possibility (page 97, RMS, 2nd edition):

Do limited backwards step-down variable selection if parsimony is more important than accuracy.

This, however, is only to be done in the context of an already well-specified model. It is the last step before the "'final' model."

As this paper linked from the related question emphasizes, the variable selection in GETS must begin with an already well-specified model:

1. The search should start from a congruent statistical model to ensure that selection inferences are reliable. Problems such as residual autocorrelation and heteroscedasticity not only reveal mis-specification. They can deliver incorrect coefficient standard errors for test calculations. Consequently, the algorithm must test for model mis-specification in the initial general model.

This is a good deal different from many questions on this site, where those without much evident statistical background often seem to want a plug-and-play approach to the entire problem. They start with some type of multiple regression, give little thought to underlying subject matter, data transformations, the peculiar problems in time series, or the like, and want to determine automatically "what are the most important variables"?

Also, GETS seems inherently inapplicable to the $p>n$ setting, where so much of the difficulty (and interest, and poor statistical technique) in variable selection arises. Although time series are outside my expertise, I suspect that large time series from which autocorrelations have been removed effectively provide $n\gg p$, with many degrees of freedom left. I also wonder (without any solid knowledge of time series) whether removing time-based autocorrelations may in practice help minimize other sources of non-orthogonality among predictors.

The various flavors of GETS pay a good deal of attention to tradeoffs between Type I and Type II errors in the steps of the reduction of the initial well-specified model to a reduced form, which may obviate shrinkage corrections (or include them implicitly in the estimates of the reduced model).