Suppose we have the linear regression model $$y=\beta_0+ \beta_1x_1 + \beta_2x_2 + \beta_3x_3 + \beta_4x_4 +\epsilon$$, where $x_3$ and $x_4$ are dummy variables. We first check the significant for $x_2$, P-value is high so we drop it. Then we test for $x_3$ and $x_4$ and we also drop them. Last we test $x_1$ and we keep it. Then our model becomes $y=\beta_0+ \beta_1x_1$. So I am wondering if we change the order in which we checked the regressors, then whether the final model we get becomes different?

I think directly switch the order may not change the final thing, but I am not sure. Does someone have some ideas? Thanks in advance.

  • 2
    $\begingroup$ +1 It's a good question, but it might be worth commenting that this is not a good procedure to follow when fitting a regression model. $\endgroup$
    – whuber
    Commented Dec 21, 2020 at 12:25
  • 1
    $\begingroup$ As @whuber stated, it was never more than a rumor that dropping so-called "insignificant" variables is a good idea. Spend your effort on model specification, not model selection. If your sample size does not allow you to fit the model you use subject matter expertise to pre-specify, use unsupervised learning to reduce the dimensionality of predictors first. $\endgroup$ Commented Dec 21, 2020 at 13:06

1 Answer 1


Seems like you are implicitly describing something similar to the the backward stepwise selection method. Here you:

  1. first start with the full model with $p$ regressors and calculate the test-SSR;
  2. calculate the $p-1$ models with $p-1$ regressors, each time leaving a different regressor out, and select the model with the best test-SSR improvement;
  3. repeat 2) with the model previously chosen, updating $p \leftarrow p - 1$;

A different approach is the the forward stepwise selection method, where you do the same comparison between test-SSR improvement, but start with a single regressor and work up to $p$. The advantage here is that one can use it even if $n<p$, i.e. observations are fewer than regressors.

The two approaches are done when best subset selection is not computationally possible, i.e. calculating the test-SSR for all possible combinations. However, both backwards and forwards stagewise selection is not guaranteed to find the best model containing a subset of $p$.

Obviously, here we are using (cross-validated) test-SSR instead of significance. Other common measures are AIC, BIC, adjusted $R^2$. Doing this with significance seems to lead to a multiple testing problem, i.e. like a t-test done multiple times on different regressors instead a single F-test.

(I know it doesn't explicitly answer your question, I posted this as an alternative to your approach and maybe also to highlight some potential problems.)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.