Which order we should use to drop insignificant regressor in a linear regression model?

Question

Suppose we have the linear regression model $$y=\beta_0+ \beta_1x_1 + \beta_2x_2 + \beta_3x_3 + \beta_4x_4 +\epsilon$$, where $x_3$ and $x_4$ are dummy variables. We first check the significant for $x_2$, P-value is high so we drop it. Then we test for $x_3$ and $x_4$ and we also drop them. Last we test $x_1$ and we keep it. Then our model becomes $y=\beta_0+ \beta_1x_1$. So I am wondering if we change the order in which we checked the regressors, then whether the final model we get becomes different?

I think directly switch the order may not change the final thing, but I am not sure. Does someone have some ideas? Thanks in advance.

Answer 1

Seems like you are implicitly describing something similar to the the backward stepwise selection method. Here you:

1. first start with the full model with $p$ regressors and calculate the test-SSR;
2. calculate the $p-1$ models with $p-1$ regressors, each time leaving a different regressor out, and select the model with the best test-SSR improvement;
3. repeat 2) with the model previously chosen, updating $p \leftarrow p - 1$;

A different approach is the the forward stepwise selection method, where you do the same comparison between test-SSR improvement, but start with a single regressor and work up to $p$. The advantage here is that one can use it even if $n<p$, i.e. observations are fewer than regressors.

The two approaches are done when best subset selection is not computationally possible, i.e. calculating the test-SSR for all possible combinations. However, both backwards and forwards stagewise selection is not guaranteed to find the best model containing a subset of $p$.

Obviously, here we are using (cross-validated) test-SSR instead of significance. Other common measures are AIC, BIC, adjusted $R^2$. Doing this with significance seems to lead to a multiple testing problem, i.e. like a t-test done multiple times on different regressors instead a single F-test.

(I know it doesn't explicitly answer your question, I posted this as an alternative to your approach and maybe also to highlight some potential problems.)