28
$\begingroup$

Let's consider for example a linear regression model. I heard that, in data mining, after performing a stepwise selection based on the AIC criterion, it is misleading to look at the p-values to test the null hypothesis that each true regression coefficient is zero. I heard that one should consider all the variables left in the model as having a true regression coefficient different from zero instead. Can anyone explain me why? Thank you.

$\endgroup$
33
$\begingroup$

after performing a stepwise selection based on the AIC criterion, it is misleading to look at the p-values to test the null hypothesis that each true regression coefficient is zero.

Indeed, p-values represent the probability of seeing a test statistic at least as extreme as the one you have, when the null hypothesis is true. If $H_0$ is true, the p-value should have a uniform distribution.

But after stepwise selection (or indeed, after a variety of other approaches to model selection), the p-values of those terms that remain in the model don't have that property, even when we know that the null hypothesis is true.

This happens because we choose the variables that have or tend to have small p-values (depending on the precise criteria we used). This means that the p-values of the variables left in the model are typically much smaller than they would be if we'd fitted a single model. Note that selection will on average pick models that seem to fit even better than the true model, if the class of models includes the true model, or if the class of models is flexible enough to closely approximate the true model.

[In addition and for basically the same reason, the coefficients that remain are biased away from zero and their standard errors are biased low; this in turn impacts confidence intervals and predictions as well -- our predictions will be too narrow for example.]

To see these effects, we can take multiple regression where some coefficients are 0 and some are not, perform a stepwise procedure and then for those models that contain variables that had zero coefficients, look at the p-values that result.

(In the same simulation, you can look at the estimates and the standard deviations for the coefficients and discover the ones that correspond to non-zero coefficients are also impacted.)

In short, it's not appropriate to consider the usual p-values as meaningful.

I heard that one should consider all the variables left in the model as significant instead.

As to whether all the values in the model after stepwise should be 'regarded as significant', I'm not sure the extent to which that's a useful way to look at it. What is "significance" intended to mean then?


Here's the result of running R's stepAIC with default settings on 1000 simulated samples with n=100, and ten candidate variables (none of which are related to the response). In each case the number of terms left in the model was counted:

enter image description here

Only 15.5% of the time was the correct model chosen; the rest of the time the model included terms that were not different from zero. If it's actually possible that there are zero-coefficient variables in the set of candidate variables, we are likely to have several terms where the true coefficient is zero in our model. As a result, it's not clear it's a good idea to regard all of them as non-zero.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ With the sentence "I heard that one should consider all the variables left in the model as significant instead" I meant: "I heard that one should consider all the variables left in the model as having a true regression coefficient different form zero instead" $\endgroup$ – John M Nov 3 '15 at 10:17
  • $\begingroup$ Okay; I've added the results of a simulation that speaks to that. $\endgroup$ – Glen_b Nov 3 '15 at 11:04
  • 10
    $\begingroup$ +1 I was doing the same simulations this weekend to prepare for a class on model selection methods. I obtained the same patterns of results, exploring $k=3$ to $39$ variables and using $10k$ observations. The next step is to see what a Bonferroni correction might do ... . $\endgroup$ – whuber Nov 3 '15 at 15:52
  • 7
    $\begingroup$ @whuber indeed, seeing what effect a Bonferroni would have (on a variety of aspects of the problem) was my immediate inclination on completing the above simulation as well, but it's not what people actually tend to do with stepwise so I didn't address it here. I'd be fascinated to hear you discuss model selection methods. I expect I'd learn quite a bit. $\endgroup$ – Glen_b Nov 3 '15 at 20:52
  • $\begingroup$ @Glen_b: (Quoted from your answer)This means that the p-values of the variables left in the model are typically much smaller than they would be if we'd fitted a single model even "if the one model we fit happens to be the one that generated the data, whether the true model is null or not". Can you explain the highlighted part a bit? How it can be that the p-values are smaller in the model that has the same specification as the data generating process (true model)? $\endgroup$ – shani Sep 26 '17 at 9:34
8
$\begingroup$

An analogy may help. Stepwise regression when the candidate variables are indicator (dummy) variables representing mutually exclusive categories (as in ANOVA) corresponds exactly to choosing which groups to combine by finding out which groups are minimally different by $t$-tests. If the original ANOVA was tested against $F_{p-1, n-p-1}$ but the final collapsed groups are tested against $F_{q-1, n-q-1}$ where $q < p$ the resulting statistic does not have an $F$ distribution and the false positive probability will be out of control.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I've read this in your book but I'm still grasping to understand in what ways do p-values get biased. To my understanding, the stepwise-selected model will yield the same coefficient p-value had it been selected via domain knowledge, and this is the part that confuses me. Could you please point me to a simulated example where p-value bias can be shown empirically? $\endgroup$ – Digio Jun 21 at 9:13
  • $\begingroup$ There are two ways to think about this. First, if you have a hundred candidate variables and all 100 are random vectors, you will easily find a few variables with very small p-values when computed the usual way. The "impressive" variables you find with this process have their nominal p-values computed devoid of context, not accounting for the number of candidate variables. These p-values are misleading. Second, go bad to the basic definition: p=P(finding something more extreme than one you found | $H_0$). The "more extreme" must consider context. $\endgroup$ – Frank Harrell Jun 23 at 13:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.