2
$\begingroup$

We know that due to multi-collinearity, the standard errors of beta estimates get inflated. But what is the mathematical basis to it?

I am looking for some mathematical relationship or something to explain this.

Like I understand if standard error of betas goes up, then t-statistics goes down and we might not be able to reject the null or the variables would appear non-significant.

But what is the mathematical relationship between multicolllinearity and inflation in variance of coefficients?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ When some linear combinations of predictors (including the constant) are almost zero, the variance of a coefficient for that combination will have extremely large variance (there's little-to-no information in the data about it). That "projects" onto the coefficients in your model ... every variable that appears in that poorly-determined linear combination will 'inherit' some of that indeterminacy (i.e. get a large variance for its estimated coefficient). $\endgroup$
    – Glen_b
    Commented May 3, 2015 at 3:16

1 Answer 1

Reset to default
3
$\begingroup$

Take a look at The Analysis of Market Demand (JSTOR) by Richard Stone, Journal of the Royal Statistical Society, Vol. 108, No. 3/4 (1945), pp. 286-391. I can't find an ungated link, so here's the main result.

He gives a formula for the estimated variance of OLS regressor $\beta_k$ in a regression of $y$ on $K$ variables as $$ \frac{1}{N-K}\cdot\frac{\sigma^2_y}{\sigma^2_k}\cdot\frac{1-R^2}{1-R^2_k}, $$ where $\sigma^2_y$ is the estimated variance of $y$, $\sigma^2_k$ is the estimated variance of $x_k$, $R_k^2$ is from the regression of $x_k$ on $K-1$ remaining independent variables, and $N$ is the sample size. The set of $K$ already includes a constant.

As the independent variables get more collinear, $R^2_k$ approaches one, so the variance blows up.

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ The specific value $1/(1-R^2_k)$ is called the variance inflation factor. It's 1 only when $x_k$ is orthogonal to all other predictors in the model. $\endgroup$
    – AdamO
    Commented Nov 21, 2018 at 16:01
  • $\begingroup$ Is there any intuitive reason as to why estimated variance for coefficient depends on r2 of of regressing xj on k - 1 remaining? $\endgroup$
    – amineh
    Commented Jun 23 at 14:09
  • $\begingroup$ @amineh Think about a toy bivariate case. Imagine I am trying to predict how much weight someone can curl using a barbell based on the circumference of their left and right biceps (X1 and X2). The size of the left and the right are very correlated; bigger biceps help to lift more weight, but my regression cannot tell if it's the right one or the left one since it is rare to find someone where there is a big difference in a sample. $\endgroup$
    – dimitriy
    Commented Jun 23 at 16:09
  • $\begingroup$ This makes the bicep coefficients be big but have high variance. If you were to do a joint test that both are zero or equal, you would reject the first null but not the second. But each coefficient’s own CI would likely include zero. If weightlifting is unfamiliar, think of jumping for height or distance and leg size. $\endgroup$
    – dimitriy
    Commented Jun 23 at 16:09
  • $\begingroup$ Thanks @dimitriy. Great example. It made me a bit more confused though. let's say all example are perfectly symmetric. Then including both biceps, yields beta exactly half of what we get if only right bicep in the model. Then the range would also be half which means less $\endgroup$
    – amineh
    Commented Jun 23 at 20:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.