7
$\begingroup$

I first ran a regression with all the variables and then ran the regression again with only significant variables (or the variables of interest).

One of the variables in the 2nd regression is highly correlated with another variable of interest, though both of them are significant but are highly correlated. And this is causing the sign of the coefficient of other variables to go opposite way (not as per expectation or theory).

My question is that can we still remove one of those (correlated) significant variables? It makes the whole result come as per the expectations.

Thanks

Improve this question
$\endgroup$
11
  • 1
    $\begingroup$ To begin with, removing all insignificant variables is a bad idea. Two of them could be correlated with each other and "mask" each others' influence, so it may be that both are insignificant but at least one of them is actually needed. Why do you want to remove variables in the first place? This is by no means mandatory. Removing significant variables is even worse by the way. $\endgroup$
    – Christian Hennig
    Jun 25, 2021 at 9:05
  • $\begingroup$ The number of observations do not allow keeping all the variables. Here is the case: I am running a panel regression with covariates of interest and country dummies. The number of observations are only 130. So, what I thought is that, first run a regression with all of the variables and then remove the insignificant ones (at least from country dummies) to safeguard the degrees of freedom. $\endgroup$
    – Kaptain Abid
    Jun 25, 2021 at 9:10
  • 1
    $\begingroup$ In that case techniques such as backward selection removing one variable at a time are better then removing all insignificant variables. More modern techniques such as the Lasso can improve on that. $\endgroup$
    – Christian Hennig
    Jun 25, 2021 at 9:18
  • 1
    $\begingroup$ Can't comment on what your supervisor says without knowing all the details. $\endgroup$
    – Christian Hennig
    Jun 25, 2021 at 10:15
  • 3
    $\begingroup$ I believe you are encountering an example of Simpson's Paradox. Please read en.m.wikipedia.org/wiki/Simpson's_paradox This it is not as you expected but indeed acting accordingly to theory! $\endgroup$
    – Chris
    Jun 25, 2021 at 13:20

3 Answers 3

Reset to default
11
$\begingroup$

To just address the actual question: Significance means that there is evidence that the variable has a nonzero contribution given all other variables in the model. This means that correlation is not a valid reason to remove a significant variable, because its significance means that its contribution can not be accounted for by the other variable.

Otherwise I agree regarding the critical remarks on variable selection based on p-values in general. The Lasso may be better here. Also, if you choose analyses based on what you expect from theory, your analyses will be invalid because you bias them in order to fit your theory.

Improve this answer
$\endgroup$
2
  • $\begingroup$ "Also, if you choose analyses based on what you expect from theory, your analyses will be invalid because you bias them in order to fit your theory." This is correct. But what to do, when lets say, the descriptive analysis tells that A happens whenever there is B. And in the results, it comes out the opposite. Then you see why is this sign not as per the "expectation", you see the reason. Can we not address the issue? Will it not be that we let the wrong relationship come up when the data actually shows the opposite? $\endgroup$
    – Kaptain Abid
    Jun 25, 2021 at 9:36
  • 3
    $\begingroup$ In that case there seems to be confounding or interaction between your variables and you may want to explore the causal structure between your variables in more detail, as was suggested in the answer by Robert Long. $\endgroup$
    – Christian Hennig
    Jun 25, 2021 at 10:51
14
$\begingroup$

Please do not choose variables for a regression model based on p-values.

Also please do not choose variables so that you achieve results that "meet expectations".

If you include variables in your model to begin with, this was presumably because they were identified as possible confounders or as variables that were not associated with your main exposure but associated with the outcome (competing exposures). These are GOOD reasons, and p-values are not part of the decision making.

Of course it is important not to over-adjust for possible confounding, and to avoid things like conditioning on a collider. The best way to do this in my opinion is by considering the possible causal relations between the variables using a causal diagram or DAG, which will then inform you of the minimally sufficient set of variables to condition on. A good free online too for doing this is http://www.dagitty.net/

Also, this answer may help you understand the pitfalls of not choosing variables in a principled manner.
How do DAGs help to reduce bias in causal inference?

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Does this answer your question ? If so please consider marking it as the accepted answer. If not, please let us know why. Also, if you haven't already, please consider upvoting it. $\endgroup$
    – Robert Long
    Jul 24, 2021 at 12:03
1
$\begingroup$

And this is causing the sign of the coefficient of other variables to go opposite way (not as per expectation or theory).

Basically (i.e. this is an oversimplification), this means that the effect of the second variable is negative, once the effect of the first is controlled for. For example, suppose you're doing a regression on life span, and one of your variables is diabetes diagnosis, and another is use of insulin. You'll probably find that insulin use is negatively correlated with life span. But when you include diabetes diagnosis, the effect will probably become positive. That's because someone who has diabetes and is using insulin has a lower life expectancy when compared to the general population, but a higher life expectancy when compared to someone with diabetes but isn't taking their insulin shots.

My question is that can we still remove one of those (correlated) significant variables? It makes the whole result come as per the expectations.

Well, what the goal of the regression? To gain insight into reality, or to come up with numbers that accord with your expectations? If having a negative coefficient really bothers you, you can look at what happens when you do PCA and do regression on the principal components rather than the original variables. In my example above, "has diabetes and doesn't take insulin" would probably be the first component, and "takes the medically appropriate level of insulin" the second. Looking at what the components come out to be and what their coefficients may give you more insight into what's going on, and result in the coefficients "making sense" more.

The lesson here is that properly interpreting regression is complicated, and involves more than just looking at the coefficients. Just saying "the coefficient for a variable represents how much effect that variable has on the response variable" is a simplification that sometimes is very misleading.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.