I'm not sure if this is "step-wise" model selection, but here is what I'm doing

  1. Decide a handful of models through exploratory data analysis.
  2. Fit the models to the data, and calculate their AIC.
  3. Pick the model with the lowest AIC score.

I've read that step-wise AIC is "primarily problematic for inference", but don't understand the rationale behind this claim. While I know there are differences between inference and prediction, I'm unaware of the restrictions they put on analytical procedures.

Is my model selection procedure problematic for inference? for prediction? Why?

  • $\begingroup$ If the "exploratory data analysis" involves the test data in any way then there is potential for optimistic bias in the performance estimate for the final model. This potential is not necessarily negligible. It also means your work is not easily reproducible. $\endgroup$ – Dikran Marsupial May 11 '19 at 16:08
  • $\begingroup$ @DikranMarsupial To be more specific, I'm doing time series analysis, and "exploratory data analysis" means deciding which of the AR/MA/ARMA models are applicable and their approximate orders by looking at the ACF/PACF plot. I suppose that's OK? $\endgroup$ – nalzok May 11 '19 at 16:10
  • $\begingroup$ ANY choice you make about your model that depends on the data used to evaluate it may introduce a bias. That bias may be negligible, or it may not, my work has shown that some of these biases thought to be negligible (in classification) turned out not to be. The more choices, the larger the bias, likewise the higher the variance of the performance estimate, the more problematic it is likely to be. $\endgroup$ – Dikran Marsupial May 11 '19 at 16:28
  • $\begingroup$ Likewise, if you use a penalisation method, it may not account for the "degrees of researcher freedom" introduced by the "exploratory data analysis" (essentially because the choice may be influenced by factors that exploit the noise in the data, rather than the underlying structure). Again, often this is entirely negligible, but sometimes it isn't. $\endgroup$ – Dikran Marsupial May 11 '19 at 16:32
  • $\begingroup$ @DikranMarsupial I see, would you suggest skipping step 1, and using ARMA($p$, $q$) for all $p, q < 5$ as the candidates in step 2 then? This is contrary to these answers which appears to advocate looking at (P)ACF before fitting the models, so I'm a little confused... $\endgroup$ – nalzok May 11 '19 at 16:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.