What happens when GLM (in R) is given 2 correlated variables?

Question

I have this data where I chose the feature as - Amount of credit and the variable to be predicted is the Credit Rating (good or bad).

I ran the logistic regression and I got an accuracy of around 67%. And my model summary was as follows -

Then I created a new (synthetic variable) as RandomAmtofCredit= Amount of credit + rnorm().

These two variables are correlated with value of 1. Here too I get almost the same accuracy; but the model summary is very confusing

My question is; how is the prediction same when both the values are almost similar and the seem to be almost cancelling out each other. (0.03540 and -0.03578).

And how does the glm function handle these correlations in general?

Answer 1

Since you added random noise of low relative RMS, they are not correlated with value 1; merely 0.9999 or so. Correlation at value 1 would not give a well-posed mathematical regression problem.

Note also that the estimates for Amount.of.Credit and RandomAmtofCredit coefficients add up to almost zero, and the difference of the coefficients is equal (in magnitude) to the coefficient of Amount.of.Credit in the original regression. This isn't a coincidence; the noise you added happened to be mildly negatively correlated with the outcome variable, and the estimate for the original variable is offset to compensate on average.

This is not an artifact of GLM; it happens in linear models too and since GLMs behave like linear models when they are nearly converged, exactly the same thing happens for GLMs. Since you asked, GLMs are sometimes solved by iteratively reweighted least squares; simply, solving the approximate LM over and over until you converge to the GLM solution.

This problem can be either solved or worked around, depending on your philosophy, by use of regularization methods like L2/Ridge/Tikhonov regularization (which would shrink the noise effect on both estimates toward 0) or L1/LASSO regularization (which would with high probability remove the RandomAmtofCredit variable). These methods will also allow solution of regressions with singularities, i.e. when the correlation of some two $X_\cdot$ variables is 1.