Collinear independent variables can have undesirable effects on the interpretation of coefficients in a linear model. Indeed, for two perfectly-correlated predictors, the coefficients are not uniquely determined, leaving a single degree of freedom by which they may vary.
However, adding in even a little bit of randomness gives solutions to the OLS equations that are not too far from assigning the correlated independent variables equal weight.
An example. The following (python) code creates a simple linear relationship between a single independent variable and a single dependent variable. It then copies the independent variable to generate a second independent variable and adds a tiny amount of independent gaussian noise to both. Thus, the second variable is nearly-perfectly correlated with the first variable. When we do this a number of times and plot the resulting coefficients, they tend to center around 2.5 and 2.5:
import numpy as np
import matplotlib.pyplot as plt
import sklearn.linear_model
coefs = [] # to hold the coefficient of all of the OLS fits
for i in range(1000): # run OLS a bunch to see what the coefficients do
X = np.linspace(0,1,100) # some independent variable
y = 5*X + 4 # some dependent variable
X2 = X # a new independent variable that is correlated with the first
XX = (
np.stack((X,X2)).T + # stack the independent variables
np.random.normal(0,0.01,(100,2) # add noise
)
lr = sklearn.linear_model.LinearRegression()
lr.fit(XX,y)
coefs.append(lr.coef_) # get the coefficients of an OLS linear regression
coefs = np.array(coefs)
plt.scatter(coefs[:,0], coefs[:,1])
plt.show()
Why am I relatively unlikely to get, say, an X1
coefficient of -105 and an X2
coefficient of 110? Those add up to 5, but there is something pushing the results toward 2.5, 2.5. What is the intuition behind this phenomenon, and what practical implications does this have when faced with collinear independent variables?
UPDATE:
Altering the amount of Gaussian noise to have a much smaller variance ($\sigma = 0.00000001$ as opposed to $\sigma=0.01$ in the example above):
And $\sigma=0.0000000000000001$:
UPDATEUPDATE:
Interest in how the amount of added noise plays into this phenomenon encouraged me to make the following graph. I ran the above experiment for various quantities of added noise, from $2^{1}$ to $2^{-60}$ on an exponential scale. I then measured the std of the X1 coefficient in the above experiment for each of those trials. Results are given here (should read $-\log_2$ on the x-axis):
I suspect that at very very small values of added noise, there are numerical problems due to limits of the computer architecture. Accounting for that, it seems from this graph that the distribution of OLS coefficients for X1 and X2 converges to a Gaussian distribution with mean 2.5 and std 0.25 or so. Thus, this seems like "Why not coefficients (-105,110)?" is still open.
Zooming in on the weird behavior for very small values:
Which seems like it's doing a sane thing. That doesn't mean that it's not floating-point errors, but it's not clear that that is what's happening.
X
in the definition ofy
and (b) multiplyingX2
by different factors. I believe the initial rise to a constant sd is an artifact of your software: it does not occur inR
'slm
function, for instance. This latter fact is a strong clue about what's going on. $\endgroup$