Clarification on Danger of Inference on Predictors for Time Series Regression

Question

I am reading the excellent book "Forecasting: Principles and Practice" and in chapter 7, section 5 there is the small section.

Beware of inference after selecting predictors

We do not discuss statistical inference of the predictors in this book (e.g., looking at p -values associated with each predictor). If you do wish to look at the statistical significance of the predictors, beware that any procedure involving selecting predictors first will invalidate the assumptions behind the p-values. The procedures we recommend for selecting predictors are helpful when the model is used for forecasting; they are not helpful if you wish to study the effect of any predictor on the forecast variable.

Would someone give a bit more detail on the bolded sentence (emphasis mine) above? Why would applying a procedure to select predictors, like AIC or BIC, invalidate the assumptions behind p-values?

Answer 1

This is absolutely not specific to time series analysis (so I took the liberty of editing your tags). Any model selection procedure will try to select predictors that make the model "better". But the mathematical theorems used in calculating p-values - specifically, theorems about the distribution of certain statistics under the null hypothesis do not make allowance for this kind of selection. Essentially, the null hypothesis never included a selection step.

You can run a quick simulation of this effect yourself. Simulate $n$ observations data points, and a $n\times k$ matrix of predictors that are completely unrelated to the observations. Run a regression, and record the $p$ values of your coefficients' $t$ tests. Do this many times. The $p$ values should be approximately uniformly distributed.

Now redo the experiment, but in each run do some model selection beforehand. Stepwise by significance or AIC, forward or backward, all subsets, whatever - it does not matter. Again record the $p$ values of the remaining predictors.

You will find that they are biased low, and badly so - but since your original predictors were completely unrelated to your simulated outcome, this should not happen. The distribution of the $p$ values is not uniform any more. So you cannot use low $p$ values to reject the null hypothesis any more. The model selection step invalidated the $p$ values. (And note that there is nothing about time series analysis anywhere.)

This is discussed at length at many places, e.g., Is p-value essentially useless and dangerous to use? (see also Gelman's pieces and the ASA statement on $p$ values cited there), or How much do we know about p-hacking "in the wild"?