1
$\begingroup$

I have two regression models. The 2nd model is obtained by removing one variable from the first. The removed variable had high multicollinearity although very low p-value. (A variable RGDP was showing high correlation with another one, MS. So I removed MS from the model).

enter image description here

My confusion is, normally multicollinearity causes p values to increase making the variable no longer significant.

enter image description here

The partial residual plot of Full Model :

enter image description here

$\endgroup$

3 Answers 3

4
$\begingroup$

A few things: First, a single variable cannot have high collinearity (or low collinearity either). It's a feature of groups of variables. Here RDGP and MS are highly related. You could have removed either one.

Second, p values are not a good reason to keep or remove variables, at least, not by themselves.

Third, while non-perfect collinearity does not violate the rules of regression, it does make interpretation tricky. The full model is giving the effect of each variable controlling for the others. You haven't told us what the variables are, but you have to ask if MS, controlling for RDGP, makes any sense.

Fourth, collinearity can make it so that small changes in the input data result in huge changes in the parameter estimates. If your only goal is prediction, that doesn't matter. But if you are trying to explain things, it does. In one of his books, David Belsley shows an example where a change in the third significant figure of the data flips the signs of the parameter estimates (and they are significant in both directions!) That could be very problematic for interpretation.

Fifth, I'm not sure why you think that collinearity "normally" increases p value. It certainly can do that, but it doesn't have to do so, by any means.

Finally, there are other ways to deal with collinearity, you don't have to drop a variable. E.g. You can combine them, or use ridge regression.

$\endgroup$
2
  • 1
    $\begingroup$ "Fifth, I'm not sure why you think that collinearity "normally" increases p value. It certainly can do that, but it doesn't have to do so, by any means." I think the logic behind this is that if a variable $x_1$ has a certain influence and it is highly correlated with another variable $x_2$, the t-test testing for influence of $x_1$ given all other variables in the model may have a large p-value due to much of the information in $x_1$ being represented by $x_2$, which is still there when removing $x_1$. One can construct opposite examples, but it seems less "natural". $\endgroup$ Commented Oct 21, 2023 at 9:59
  • $\begingroup$ @ChristianHennig OK, that makes sense. $\endgroup$
    – Peter Flom
    Commented Oct 21, 2023 at 10:08
2
$\begingroup$

It would help if you printed the $R^2$ and the tables from both models. I would expect the t-stat of $ms$ to increase further once $RGDP$ is removed. However, it is possible that there is a special relationship between the 2 and the dependent, in a way that $RGDP$ helps filter out some element out of $ms$ making it more meaningful in explaining the dependent variable. In that case, you almost certainly need it.

You could do a Likelihood Ratio test to see if including the extra variable significantly adds to the explanatory power of the model. If it does not, then you can be quite confident that getting rid of it is the right decision. Personally, I am stricter than that and would demand that the increase in $R^2$ is "more" than simply statistically significant; i.e. it really makes a difference to the model, according to whatever my criteria in each case are.

In case you conclude that $RGDP$ adds value on top of the other variables, try to think of theoretically why this could be and maybe consider if it is possible to combine it with $ms$ in one variable that could give you all that both do, or come up with a less correlated combination between the two.

$\endgroup$
1
  • $\begingroup$ "You could do a Likelihood Ratio test to see if including the extra variable significantly adds to the explanatory power of the model" - if the t-test is significant, this should also be significant and not add information. "If it does not, then you can be quite confident that getting rid of it is the right decision." Not so. Even insignificant variables shouldn't be removed unless there are strong reasons such as not having enough observations to estimate the parameters with reasonable precision. $\endgroup$ Commented Oct 21, 2023 at 9:41
2
$\begingroup$

The p-value indicates that the variable contributes significantly to the quality of the model. So in my view there is no reason to remove it. Correlation with another variable is in no way a violation of the validity of a regression model. If however there is a truly influential variable that is not in the model, this actually is a violation of the model. Generally I think that many people are far too obsessed with removing variables. If you have enough observations to estimate regression coefficients with reasonable precision, variables shouldn't be removed unless there are strong reasons (such as predictions in the future for which some variables are costly to measure, so you save money if you have a model without them). Note that data dependent removal of variables generally will invalidate standard inference on the reduced model. Correlation between variables as well as insignificant predictors or predictors with a small (if significant) influence are not a problem that justifies variable removal, as these things can be seen and results appropriately interpreted if the variable remains in the model (of course the situation is different if you have 20 observations for 15 variables and the like).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.