5
$\begingroup$
  • I wonder why general-to-specific (GETS) subset selection and particularly the Autometrics algorithm are performing well in macroeconomic modelling/forecasting.

How does Autometrics work?

  1. Doornik "Autometrics" (2009) offers a full description (see p. 7-8 for the algorithm scheme).
    The algorithm is based on stepwise subset selection with qualifications and includes diagnostic testing.
    A few main features are listed below.
  2. The general model is formulated as a linear model with
    (i) main variables,
    (ii) generous lags thereof, and in later versions also
    (iii) impulses (for impulse indication saturation (IIS) or step indication saturation (SIS) techniques - to account for possible structural breaks) and
    (iv) nonlinear transformations.
  3. The algorithm does stepwise selection starting from a general model and going down several paths.
  4. Variables may not only be removed but also added at each step (except the initial step where all the variables are included).
  5. Significance testing is used for variable removal/inclusion.
  6. When the search terminates (based on significance of variables), diagnostic testing is carried out. Poor models are removed from further consideration.

How well does Autometrics perform?

  1. Good performance of Autometrics is apparent from works of Sir David F. Hendry, Jurgen A. Doornik et al. (in the links, see works on model selection, Autometrics and related).
    Of course, independent authors' experience would be more credible than the experience of the "fathers" of Autometrics.
  2. Kock & Teräsvirta "Forecasting performances of three automated modelling techniques during the economic crisis 2007–2009" (2014) find:

[Autometrics] works well when the model is a reasonable approximation to reality, but less well when it is not. In direct forecasting, one is facing the latter situation... It appears that Autometrics may not be an appropriate tool for building models for direct multiperiod forecasting. However, it can be an excellent choice when the data-generating process is approximated well by a subset of variables in the data set of the researcher.

  1. Epprecht et al. "Comparing variable selection techniques for linear regression: LASSO and Autometrics" (2013) find that Autometrics and LASSO outperform each other in different respects and different settings.

  2. The typical setting in Hendry's own performance testing of Autometrics is when the correct model is a subset of the general model. Autometrics is shown to select a model that is either the true one or very close to it.
    Meanwhile, under tapering effects or considerable misspecification of the general model Autometrics may perform less well, e.g. as in Kock & Teräsvirta (2014) above.

Why is good performance surprising?

  1. With no shrinkage and having to select among a large number of variables (that is the setting in many of Hendry's and Doornik's works), I would anticipate GETS to fail. (E.g. Frank Harrell has condemned subset selection in several posts, e.g. 1 and 2, although the posts were not specific to macroeconomic time series.)
  2. (Question) Could the good performance of GETS and Autometrics be specific to the applications (mostly macroeconomic time series modelling),
    or is Autometrics just a really good method in a broad sense?

"LASSO/LARS vs general to specific (GETS) method" is a related question.

$\endgroup$
  • $\begingroup$ Interesting post/question. It seems to me that autometrics is very taylored to time-series macroeconomics. Consider however model selection in panel data, where there is a whole array of other issues involved, e.g. clustering choices, unobserved heterogeneity, selection, missing data, and etc. I would be very keen to see a simulation of autometrics' performance in that context. Also, in models with heterogeneous agents, autometrics is helpless. Hendry said this himself to me once. $\endgroup$ – luchonacho Nov 16 '18 at 11:30
  • $\begingroup$ @luchonacho, glad you found it interesting. I also envy your for having discussed this with Hendry himself! My gut feeling about GETS/Autometrics is skeptical, and I cannot even say why exactly, but I have met some proponents (e.g. the author of the R package gets) who are quite enthusiastic about it. $\endgroup$ – Richard Hardy Nov 16 '18 at 11:43
  • $\begingroup$ He taught me econometrics :) Autometrics is a kind of religion around here. You know, he is the original author of the cointegration concept, which he discussed with Granger, who went on and published and became famous (even to the Nobel prize level) for it, without ever acknowledging Hendry. It's sad actually. $\endgroup$ – luchonacho Nov 16 '18 at 12:24
  • 1
    $\begingroup$ @luchonacho, lucky you! Hendry should probably have been awarded the prize together with Engle and Granger. In my humble and ill-informed opinion, all three of them were giants in the field. I especially admire Granger and then Hendry, I think. $\endgroup$ – Richard Hardy Nov 16 '18 at 15:23
1
$\begingroup$

Frank Harrell does not rule out intelligent use of backward elimination. He includes as a possibility (page 97, RMS, 2nd edition):

Do limited backwards step-down variable selection if parsimony is more important than accuracy.

This, however, is only to be done in the context of an already well-specified model. It is the last step before the "'final' model."

As this paper linked from the related question emphasizes, the variable selection in GETS must begin with an already well-specified model:

  1. The search should start from a congruent statistical model to ensure that selection inferences are reliable. Problems such as residual autocorrelation and heteroscedasticity not only reveal mis-specification. They can deliver incorrect coefficient standard errors for test calculations. Consequently, the algorithm must test for model mis-specification in the initial general model.

This is a good deal different from many questions on this site, where those without much evident statistical background often seem to want a plug-and-play approach to the entire problem. They start with some type of multiple regression, give little thought to underlying subject matter, data transformations, the peculiar problems in time series, or the like, and want to determine automatically "what are the most important variables"?

Also, GETS seems inherently inapplicable to the $p>n$ setting, where so much of the difficulty (and interest, and poor statistical technique) in variable selection arises. Although time series are outside my expertise, I suspect that large time series from which autocorrelations have been removed effectively provide $n\gg p$, with many degrees of freedom left. I also wonder (without any solid knowledge of time series) whether removing time-based autocorrelations may in practice help minimize other sources of non-orthogonality among predictors.

The various flavors of GETS pay a good deal of attention to tradeoffs between Type I and Type II errors in the steps of the reduction of the initial well-specified model to a reduced form, which may obviate shrinkage corrections (or include them implicitly in the estimates of the reduced model).

$\endgroup$
  • $\begingroup$ Actually, Autometrics specifically emphasizes that it works fine with $p>n$. The initial general model in the late versions of Autometrics is by default $p>n$ as it includes $n$ orthogonal dummies for structural breaks. But I am not sure if it works well when $p>2n$ where $n$ variables are orthogonal dummies but the remaining $p-n$ variables are not orthogonal dummies. $\endgroup$ – Richard Hardy May 24 '16 at 15:47
  • $\begingroup$ I also wonder how a smart GETS like Autometrics could obviate shrinkage corrections (or include them implicitly in the estimates of the reduced model). If this is indeed the case, then GETS becomes an attractive modelling tool (because lack of explicit shrinkage in GETS is the most obvious drawback to me). $\endgroup$ – Richard Hardy May 24 '16 at 15:55
  • $\begingroup$ Any thoughts regarding my comments? $\endgroup$ – Richard Hardy Nov 16 '18 at 11:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.