# Why are p-values misleading after performing a stepwise selection?

Let's consider for example a linear regression model. I heard that, in data mining, after performing a stepwise selection based on the AIC criterion, it is misleading to look at the p-values to test the null hypothesis that each true regression coefficient is zero. I heard that one should consider all the variables left in the model as having a true regression coefficient different from zero instead. Can anyone explain me why? Thank you.

after performing a stepwise selection based on the AIC criterion, it is misleading to look at the p-values to test the null hypothesis that each true regression coefficient is zero.

Indeed, p-values represent the probability of seeing a test statistic at least as extreme as the one you have, when the null hypothesis is true. If $$H_0$$ is true, the p-value should have a uniform distribution.

But after stepwise selection (or indeed, after a variety of other approaches to model selection), the p-values of those terms that remain in the model don't have that property, even when we know that the null hypothesis is true.

This happens because we choose the variables that have or tend to have small p-values (depending on the precise criteria we used). This means that the p-values of the variables left in the model are typically much smaller than they would be if we'd fitted a single model. Note that selection will on average pick models that seem to fit even better than the true model, if the class of models includes the true model, or if the class of models is flexible enough to closely approximate the true model.

[In addition and for basically the same reason, the coefficients that remain are biased away from zero and their standard errors are biased low; this in turn impacts confidence intervals and predictions as well -- our predictions will be too narrow for example.]

To see these effects, we can take multiple regression where some coefficients are 0 and some are not, perform a stepwise procedure and then for those models that contain variables that had zero coefficients, look at the p-values that result.

(In the same simulation, you can look at the estimates and the standard deviations for the coefficients and discover the ones that correspond to non-zero coefficients are also impacted.)

In short, it's not appropriate to consider the usual p-values as meaningful.

I heard that one should consider all the variables left in the model as significant instead.

As to whether all the values in the model after stepwise should be 'regarded as significant', I'm not sure the extent to which that's a useful way to look at it. What is "significance" intended to mean then?

Here's the result of running R's stepAIC with default settings on 1000 simulated samples with n=100, and ten candidate variables (none of which are related to the response). In each case the number of terms left in the model was counted: Only 15.5% of the time was the correct model chosen; the rest of the time the model included terms that were not different from zero. If it's actually possible that there are zero-coefficient variables in the set of candidate variables, we are likely to have several terms where the true coefficient is zero in our model. As a result, it's not clear it's a good idea to regard all of them as non-zero.

• With the sentence "I heard that one should consider all the variables left in the model as significant instead" I meant: "I heard that one should consider all the variables left in the model as having a true regression coefficient different form zero instead" – John M Nov 3 '15 at 10:17
• Okay; I've added the results of a simulation that speaks to that. – Glen_b -Reinstate Monica Nov 3 '15 at 11:04
• +1 I was doing the same simulations this weekend to prepare for a class on model selection methods. I obtained the same patterns of results, exploring $k=3$ to $39$ variables and using $10k$ observations. The next step is to see what a Bonferroni correction might do ... . – whuber Nov 3 '15 at 15:52
• @whuber indeed, seeing what effect a Bonferroni would have (on a variety of aspects of the problem) was my immediate inclination on completing the above simulation as well, but it's not what people actually tend to do with stepwise so I didn't address it here. I'd be fascinated to hear you discuss model selection methods. I expect I'd learn quite a bit. – Glen_b -Reinstate Monica Nov 3 '15 at 20:52
• @Glen_b: (Quoted from your answer)This means that the p-values of the variables left in the model are typically much smaller than they would be if we'd fitted a single model even "if the one model we fit happens to be the one that generated the data, whether the true model is null or not". Can you explain the highlighted part a bit? How it can be that the p-values are smaller in the model that has the same specification as the data generating process (true model)? – shani Sep 26 '17 at 9:34

An analogy may help. Stepwise regression when the candidate variables are indicator (dummy) variables representing mutually exclusive categories (as in ANOVA) corresponds exactly to choosing which groups to combine by finding out which groups are minimally different by $t$-tests. If the original ANOVA was tested against $F_{p-1, n-p-1}$ but the final collapsed groups are tested against $F_{q-1, n-q-1}$ where $q < p$ the resulting statistic does not have an $F$ distribution and the false positive probability will be out of control.