# Can we remove significant variables in a regression?

I first ran a regression with all the variables and then ran the regression again with only significant variables (or the variables of interest).

One of the variables in the 2nd regression is highly correlated with another variable of interest, though both of them are significant but are highly correlated. And this is causing the sign of the coefficient of other variables to go opposite way (not as per expectation or theory).

My question is that can we still remove one of those (correlated) significant variables? It makes the whole result come as per the expectations.

Thanks

• To begin with, removing all insignificant variables is a bad idea. Two of them could be correlated with each other and "mask" each others' influence, so it may be that both are insignificant but at least one of them is actually needed. Why do you want to remove variables in the first place? This is by no means mandatory. Removing significant variables is even worse by the way. Jun 25, 2021 at 9:05
• The number of observations do not allow keeping all the variables. Here is the case: I am running a panel regression with covariates of interest and country dummies. The number of observations are only 130. So, what I thought is that, first run a regression with all of the variables and then remove the insignificant ones (at least from country dummies) to safeguard the degrees of freedom. Jun 25, 2021 at 9:10
• In that case techniques such as backward selection removing one variable at a time are better then removing all insignificant variables. More modern techniques such as the Lasso can improve on that. Jun 25, 2021 at 9:18
• Can't comment on what your supervisor says without knowing all the details. Jun 25, 2021 at 10:15
• I believe you are encountering an example of Simpson's Paradox. Please read en.m.wikipedia.org/wiki/Simpson's_paradox This it is not as you expected but indeed acting accordingly to theory! Jun 25, 2021 at 13:20

To just address the actual question: Significance means that there is evidence that the variable has a nonzero contribution given all other variables in the model. This means that correlation is not a valid reason to remove a significant variable, because its significance means that its contribution can not be accounted for by the other variable.

Otherwise I agree regarding the critical remarks on variable selection based on p-values in general. The Lasso may be better here. Also, if you choose analyses based on what you expect from theory, your analyses will be invalid because you bias them in order to fit your theory.

• "Also, if you choose analyses based on what you expect from theory, your analyses will be invalid because you bias them in order to fit your theory." This is correct. But what to do, when lets say, the descriptive analysis tells that A happens whenever there is B. And in the results, it comes out the opposite. Then you see why is this sign not as per the "expectation", you see the reason. Can we not address the issue? Will it not be that we let the wrong relationship come up when the data actually shows the opposite? Jun 25, 2021 at 9:36
• In that case there seems to be confounding or interaction between your variables and you may want to explore the causal structure between your variables in more detail, as was suggested in the answer by Robert Long. Jun 25, 2021 at 10:51

Please do not choose variables for a regression model based on p-values.

Also please do not choose variables so that you achieve results that "meet expectations".

If you include variables in your model to begin with, this was presumably because they were identified as possible confounders or as variables that were not associated with your main exposure but associated with the outcome (competing exposures). These are GOOD reasons, and p-values are not part of the decision making.

Of course it is important not to over-adjust for possible confounding, and to avoid things like conditioning on a collider. The best way to do this in my opinion is by considering the possible causal relations between the variables using a causal diagram or DAG, which will then inform you of the minimally sufficient set of variables to condition on. A good free online too for doing this is http://www.dagitty.net/

Also, this answer may help you understand the pitfalls of not choosing variables in a principled manner.
How do DAGs help to reduce bias in causal inference?