When fitting a multiple linear regression model, if one factor is insignificant, should I refit another model?

Question

Let's say, I have:

$y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 x_3$

I fit a multiple linear regression (MLR) model (lm() command) in R, and see a very large $p$-value for $\beta_1$ (say 0.5). Now, if I leave out $x_1$ and fit another model:

$y = \beta_0 + \beta_2 x_2 + \beta_3 x_3$

I see $\beta_2$ and $\beta_3$, and consequently, their significance change. I'm thinking it should make more sense to fit the second one and the first one is probably faulty, because of an insignificant effect in the model. So if I want to see if there is really a linear relationship between $x_2$ or $x_3$ and $y$, I should leave the other variable out, right?

Answer 1

No. Variables should not be dropped from regression models based on whether they are significant, but rather based on theory. As I note here, thresholds to compare p-values to in order to determine significant are merely heuristics. If a particular variable does not meet that threshold, it does not mean the variable should necessarily be rejected. The coefficient on the insigifnificant variable might be statistically indistinguisible from zero, but does not necessarily mean that the coefficient is zero.

That being said, you should only include the variable in your multiple regression if it makes sense theoretically to do so. That is, even if it is insignificant, you will include mileage as a variable explaining the price of a car. This makes sense theoretically.

With both of the above considerations, keeping insignificant variables in the regression allows you to control for any alternative explanations driving variation in your outcome variable. Without it, your coefficient estimates on the other variables (such as those in your second model) will be biased.

Finally, here are a number of other good responses to the same question: [1] and [2]

Answer 2

Insignificance means that the data is compatible with a model in which the regression parameter b1 for x1 is zero. However, it is also compatible with models in which the regression parameter is nonzero (I guess that your estimated b1 wasn't exactly zero, so there's another value that is also compatible with the data; actually the estimated b1 is the best guess from the data and as such better than zero, so it's more likely than not that you make your model worse by kicking it out).

Particularly, having an insignificant parameter in the model in no way means that the model would be "faulty", as the original model allows for b1=0. There is therefore hardly any reason to throw away the b1.x1. Obviously nobody stops you from leaving it in the model and saying that there is no evidence that it is needed (although it still may be better to leave it in).

As long as you have enough observations (and with three variables you really don't need that many) and the x1 isn't strongly correlated with one of the other variables, the only reason that I see kicking it out is because you may want to predict future observations without having to measure x1 because it may be expensive or tedious to measure. If none of these things apply, leave it in.