2
$\begingroup$

I have read time and again that strong collinearity between regressors in OLS regression can result in inaccurate estimates for individual coefficients. To see this action, I wrote the following code in Python :

import numpy as np
import matplotlib.pyplot as plt

num_samples = 200

# Generate correlated regressors
mu = np.array([5.0, 0.0, 3.0])

# Choose covariance matrix so that x0 and x1 are strongly correlated
cov = np.array([
        [  3.40, 2.75, 0.01],
        [  2.75, 2.30, 0.02],
        [  0.01, 0.02, 2.40]
    ])

rng = np.random.default_rng()
X = rng.multivariate_normal(mu, cov, size=num_samples)

# Extract regressors
x0 = X[:,0]
x1 = X[:,1]
x2 = X[:,2]

y = 5.12*x0 + 2.34*x1 + 3.12*x2
coeff, res , rank, s = np.linalg.lstsq(X, y)

The correlation between x0, x1 and x2 is shown in scatter-plots below (Note that x0 and x1 are strongly correlated)

enter image description here

enter image description here

enter image description here

The OLS coefficients were :

array([5.12, 2.34, 3.12])

which is perfectly correct and there is no error in the estimation.

  • Why was strong multicollinearity between x0 and x1 not an issue here ?
  • What criteria can I use to determine whether or not multicollinearity is going to be an issue ?
$\endgroup$
5
  • 3
    $\begingroup$ The error of coefficients is relatively larger for correlated regressors, but if the noise is small and/or the sample size large then this may not be a problem. $\endgroup$ Dec 28, 2021 at 11:52
  • $\begingroup$ @BigBendRegion : The true values are 5.12, 2.34 and 3.12 (see second last line in code) $\endgroup$
    – johngreen
    Dec 28, 2021 at 14:21
  • $\begingroup$ The true values are in mu, as @BigBendRegion noted. The penultimate line displays the estimates of those values. The estimates differ from the true values: there is indeed estimation error. $\endgroup$
    – whuber
    Dec 28, 2021 at 17:46
  • $\begingroup$ @whuber : Please correct me if I am wrong. Values in mu are the mean values of x0, x1 and x2, while the OLS coefficients computed 5.12, 2.34, 3.12 are the estimates of coefficients c for the equation y=X@c. $\endgroup$
    – johngreen
    Dec 29, 2021 at 6:06
  • 1
    $\begingroup$ That looks right--I misread your code. The problem with your example is that you have introduced no error in y: it is a perfect linear combination of the columns of X. No amount of collinearity in X will create difficulties in such a circumstance. $\endgroup$
    – whuber
    Dec 29, 2021 at 16:39

1 Answer 1

4
$\begingroup$

That you have perfect estimates of all three coefficients should suggest to you that something strange has happened in the simulation, and I found that strangeness.

You have no random component to your $y$ variable. Therefore, except for possible numerical issues when it comes to doing arithmetic on a computer, your regression should have perfect estimates of the coefficients, zero-width confidence intervals, and $R^2=1$. There’s no randomness, so your estimates should be perfect.

However, if you tack on an error term like np.random.normal(0, 1, num_samples), you will see your estimates get worse.

Back to the original issue, the Gauss-Markov theorem gives conditions where the OLS estimator is unbiased, and the theorem makes no mention of feature correlation. However, feature correlation affects confidence interval width on the feature coefficients through the the variance-inflation factor (VIF). As features become more correlated, their confidence intervals widen. However, the estimates are unbiased, even when the correlation is high.

(You don’t even have to appeal to Gauss-Markov, but it uses a common set of assumptions, none of which involve feature correlation.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.