0
$\begingroup$

I'm trying to prove that multicollinearity doesn't introduce bias into a multiple linear regression model, but my proof seems to indicate the opposite.

If we represent the model as $$y = \hat \beta_0 +\hat \beta_1x_1 +\hat \beta_2x_2+\hat \beta_2x_2+\hat \beta_3x_3 +\hat \beta_4x_4+\hat \beta_5x_5 + \epsilon $$

Assuming the measurements are statistically independent: $$E(\hat \beta_1)= cov(y, x_1)/var(x_1) = cov(x_1, \beta_0 + \beta_1x_1 + \beta_2x_2+ \beta_2x_2+ \beta_3x_3 + \beta_4x_4+ \beta_5x_5 + \epsilon) = 0 + \beta_1 + \beta_2cov(x1, x2)/var(x1)+ \beta_3cov(x1, x3)/var(x1) + \beta_4cov(x1, x4)/var(x1)+ \beta_5cov(x1, x5)/var(x1) $$

I get stuck at this point, because assuming that the variables are collinear, than $cov(x_1, x_i) =$ some nonzero value and thus introducing bias?

$\endgroup$
3
  • 1
    $\begingroup$ Please take care to clearly distinguish multiple linear regression from a multilinear function by avoiding the term "multilinear model" here. You seem to be using the former, but I would take "multilinear model" to mean the latter. $\endgroup$
    – Galen
    Commented Mar 5, 2022 at 21:41
  • $\begingroup$ oh yes thank you! $\endgroup$
    – RidgeAllen
    Commented Mar 5, 2022 at 21:50
  • 2
    $\begingroup$ Your formula for the expectation of the estimate is not correct: you have applied the formula for univariate regression, not multivariate regression. The easiest way forward is to write down any correct formula and note that it assumes nothing about the covariance structure of the explanatory variables. Emulate the analysis at stats.stackexchange.com/a/333357/919, for instance. (It implicitly assumes the design matrix is of full rank, but even that assumption can be eliminated by employing generalized inverses). $\endgroup$
    – whuber
    Commented Mar 5, 2022 at 22:46

1 Answer 1

1
$\begingroup$

Stick to two predictors for the moment, and suppose we want to minimise $\sum_i (Y_i-\beta_0-\beta_1X_1-\beta_2X_2)^2$. Differentiating with respect to the $\beta$s gives $$\begin{align}\sum_i Y_i-\hat\beta_0-\hat\beta_1X_{1i}-\hat\beta_2X_{2i}&=0 \\\sum_i X_{1i}(Y_i-\hat\beta_0-\hat\beta_1X_{1i}-\hat\beta_2X_{2i})&=0\\\sum_i X_{2i}(Y_i-\hat\beta_0-\hat\beta_1X_{1i}-\hat\beta_2X_{2i})&=0\end{align}$$

It's easy to see these are satisfied in expectation by the true $\beta$s $$\begin{align}E\left[\sum_i Y_i- \beta_0- \beta_1X_{1i}- \beta_2X_{2i}\right]&=0 \\E\left[\sum_i X_{1i}(Y_i- \beta_0- \beta_1X_{1i}- \beta_2X_{2i})\right]&=0\\E\left[\sum_i X_{2i}(Y_i- \beta_0- \beta_1X_{1i}- \beta_2X_{2i})\right]&=0\end{align}$$ and that this doesn't depend at all on the covariance of the $X$s.

In matrix notation $$\hat\beta = (X^TX)^{-1}X^TY=(X^TX)^{-1}X^T(X\beta+\epsilon)$$ so $$E[\hat\beta]=(X^TX)^{-1}X^TY=(X^TX)^{-1}X^T(X\beta+E[\epsilon])=\beta$$

What this calculation disguises is that the difference between $\beta_1$ and the coefficient $\gamma_1$ in $$E[Y|X_1]=\gamma_0+\gamma_1X_1$$ does depend on the correlation between $X_1$ and $X_2$. Co-linearity doesn't cause bias in the model you fit, but it means that the model you fit has different coefficients from a model with additional or omitted variables. If your $x$s were uncorrelated, $\beta_1$ would equal $\gamma_1$ and model selection would be much less interesting. I think some of the concern in textbooks about co-linearity comes from a misplaced expectation that in some sense $\beta_1$ and $\gamma_1$ should be the same and it's bad if they aren't.

As an additional point, the information in the data about $\beta_1$ does depend on the correlation between $X_1$ and $X_2$, so the precision of $\hat\beta_1$ is sensitive to co-linearity. If $X_1$ and $X_2$ are strongly positively correlated, $\beta_1$ will have a much larger variance than $\gamma_1$, which is sad if you want to estimate $\beta_1$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.