I've read countless posts on this site that are incredibly against the use of stepwise selection of variables using any sort of criterion whether it be p-values based, AIC, BIC, etc.
I understand why these procedures are in general, quite poor for the selection of variables. gung's probably famous post here clearly illustrates why; ultimately we are verifying a hypothesis on the same dataset we used to come up with the hypothesis, which is just data dredging. Furthermore, p-values are affected by quantities such as collinearity and outliers, which heavily skew results, etc.
However, I've been studying time series forecasting quite a bit lately and have come across Hyndman's well respected textbook in which he mentions here the use of stepwise selection to find the optimal order of ARIMA models in particular. In fact, in the
forecast package in R the well known algorithm known as
auto.arima by default uses stepwise selection (with AIC, not p-values). He also criticizes p-value based feature selection which aligns well with multiple posts on this website.
Ultimately, we should always cross validate in some way at the end if the goal is to develop good models for forecasting/prediction. However, surely this is somewhat of a disagreement here when it comes to the procedure itself for evaluation metrics other than p-values.
Does anyone have any opinions on the use of stepwise AIC in this context, but also in general out of this context? I have been taught to believe any stepwise selection is poor, but to be honest,
auto.arima(stepwise = TRUE) has been giving me better out of sample results than
auto.arima(stepwise = FALSE) but perhaps this is just coincidence.