If a variable is found with p-value greater than 0.05, why is it also taken for calculation in the regression equation Y=a+b1*X1+b2*X2?

Question

Suppose, I have performed multiple regression analysis on the following data set where X1 and X2 are independent variables and Y is the dependent variable.

And achieved the following multiple regression analysis table from where it is clear that X2 is not correlated with Y as the p-value is greater than 0.05 which is 0.108 for X2 variable

So, if p value is greater than 0.05 for a variable, then why it is considered in regression equation

Y = 2.709 + 0.763(X1) + 0.463(X2) which has been shown in any online multiple regression equation calculator.

My question is Why it is not like the following Y = 2.709 + 0.763(X1) omitting the [0.463(X2)] part from the equation, as X2 is not correlated with Y

Please help to clarify the conceptual idea....

Answer 1

One reason to keep $X_2$ in the model is (lack of) power:

The non-rejection implied by $p>0.05$ may be a type-II error, i.e., a wrong null hypothesis of irrelevance of $X_2$ that has wrongly not been rejected. Especially in situations like yours where the sample size is small, such problems may occur often.

Put somewhat differently, while the effect of $X_2$ is not statistically significant, it may still be significant from a subject matter point of view - 0.463 may be a "large" effect which we just may not be able to statistically distinguish from zero given the estimation uncertainty.

Also, note that, in any case, it would not be sound to report $Y = 2.709 + 0.763X_1$ as your fitted model. If you drop $X_2$ and reestimate your model with only $X_1$ as a regressor, the coefficient estimates will differ.

Answer 2

Short answer: the reason to "keep" X2 in the model is because the goal of regression analysis is (usually) not to come up with the "best" model of Y in some abstract sense, but to answer specific questions about how different independent variables are related to Y, and the model you ran (with a significant X1 coefficient and a non-significant coefficient for X2) is the ANSWER to one of those questions.

Long answer: Regression analysis is a tool that you can use to try and answer specific questions. There is no way to answer a question like "should this variable be in the model" in the abstract, but only "does including this variable in the model help me to answer the question I'm interested in." Usually, we run regression models to test a hypothesis about whether a specific independent variable is related to a dependent variable, after controlling for confounding variables.

Here you have run a model with two independent variables X1 and X2 and found that X2 is not significant at the 95% level, but X1 is. What that means for you depends on what your research question is. If the reason you ran the model was to see if X1 was related to Y (controlling for X2) then you have your answer: yes (at the 95% confidence level). If the reason you ran the model was to see if X2 was related to Y after controlling for X1 then you also have an answer: no, or rather "we do not have enough data to reject the null hypothesis that X2 and Y are unrelated at the 95% level after controlling for X1."

Either way, there is no reason to run the model again...since you already have the answer to your question.

If, on the other hand, you are running the model to try to generate good predictions for Y, given various X1 and X2 values, then you should probably still leave X2 in. Remember that the .05 p value cut off is arbitrary. X2 is still "significant" at the 82% level, so it might still be helpful in predicting Y.

Aside from all this, it's very dangerous to try and "fine tune" regression models - by dropping (or adding) variables and re-running the model again and again. Doing this actually violates the assumptions of significance testing. What "significant at the 95% level" actually means is that, if you replicated your sample 100 times, and there was no real difference between X and Y, you would only (wrongly) see a difference this big in 5 of those samples. But that means that if you actually run a ton of different models with different variables, then the chance of getting a "false positive" on one of them is actually going to be way more than 5%. This is called the multiple comparisons problem, and it's a big reason that statisticians don't like "stepwise" methods of specifying models (where you keep re running the model, dropping non-significant variables each time). Dropping X2 because it's non significant is basically a version of that approach.

The solution to this is to not try and fine tune models after you have run them. Come up with a good research question. Use theory and past research to decide on on a model (and a set of variables to include) that provide a good test of that question, run THAT model and report ALL of the results, whether they are significant or not.