# Which order we should use to drop insignificant regressor in a linear regression model?

Suppose we have the linear regression model $$y=\beta_0+ \beta_1x_1 + \beta_2x_2 + \beta_3x_3 + \beta_4x_4 +\epsilon$$, where $$x_3$$ and $$x_4$$ are dummy variables. We first check the significant for $$x_2$$, P-value is high so we drop it. Then we test for $$x_3$$ and $$x_4$$ and we also drop them. Last we test $$x_1$$ and we keep it. Then our model becomes $$y=\beta_0+ \beta_1x_1$$. So I am wondering if we change the order in which we checked the regressors, then whether the final model we get becomes different?

I think directly switch the order may not change the final thing, but I am not sure. Does someone have some ideas? Thanks in advance.

• +1 It's a good question, but it might be worth commenting that this is not a good procedure to follow when fitting a regression model.
– whuber
Dec 21, 2020 at 12:25
• As @whuber stated, it was never more than a rumor that dropping so-called "insignificant" variables is a good idea. Spend your effort on model specification, not model selection. If your sample size does not allow you to fit the model you use subject matter expertise to pre-specify, use unsupervised learning to reduce the dimensionality of predictors first. Dec 21, 2020 at 13:06

Seems like you are implicitly describing something similar to the the backward stepwise selection method. Here you:

1. first start with the full model with $$p$$ regressors and calculate the test-SSR;
2. calculate the $$p-1$$ models with $$p-1$$ regressors, each time leaving a different regressor out, and select the model with the best test-SSR improvement;
3. repeat 2) with the model previously chosen, updating $$p \leftarrow p - 1$$;

A different approach is the the forward stepwise selection method, where you do the same comparison between test-SSR improvement, but start with a single regressor and work up to $$p$$. The advantage here is that one can use it even if $$n, i.e. observations are fewer than regressors.

The two approaches are done when best subset selection is not computationally possible, i.e. calculating the test-SSR for all possible combinations. However, both backwards and forwards stagewise selection is not guaranteed to find the best model containing a subset of $$p$$.

Obviously, here we are using (cross-validated) test-SSR instead of significance. Other common measures are AIC, BIC, adjusted $$R^2$$. Doing this with significance seems to lead to a multiple testing problem, i.e. like a t-test done multiple times on different regressors instead a single F-test.

(I know it doesn't explicitly answer your question, I posted this as an alternative to your approach and maybe also to highlight some potential problems.)