# Removing highly correlated variables in logistic regression in r

I am developing a logistic regression model on a large dataset consisting of 15 variables and 200k observations. In initial model fitting, I find variables - "Purchase Frequency" and "Average Payment Amount" are highly correlated (GVIF values around 20) and both are significant in terms of p value. When I remove one of these two variable, the other variable becomes insignificant and also few other variables (low VIF value) becomes insignificant from significant in previous case.

How should I proceed with this?

• As @Alex mentioned this question is for Cross Validated. Nevertheless, to help you phrase you question better, it seems like you face problem of strong multicollinearity. You should research this topic a bit more UCLA's blog is a good read for start. ats.ucla.edu/stat/stata/webbooks/logistic/chapter3/… Jun 16, 2016 at 9:48

## 2 Answers

You should probably remove one of the variables (I say probably as a general case because I don't have the output - if you want a more accurate answer, post some of the residuals / results)

$p$-values aren't an 'instant win' in terms of models - having a statistically significant $p$-value on one of your variables does not guarantee that your overall model is statistically significant or useful. In fact, $p$-values can be very deceptive.

For example, consider the following simplified dataset:

Y   X   Z
0   1   1
0   2   2
0   3   4
0   6   6
0   9   8
0   2   2


You could build a 'beautiful' model with $Y = 100X - 100Z$. This model would have a great $R^2$, and really low $p$-values, but it's pretty obvious that this model isn't the best way to look at $Y$.

In the same manner, your correlated variables may be like the $X$ and $Z$ here. You should take one of them out in order to see patterns more clearly.

One way to proceed is to take a ratio of the two highly correlated variables. Considering your variables are Purchase and Payment related, am sure the ratio would be meaningful. This way you capture the effects of both, without bothering the other variables. On a separate note, if the significance and/or correlation of other variables is changing on dropping a variable, you may want to take another look at the model itself.