Why does the standard error for betas of predictor variables increases when they are highly correlated, indicating high levels of multicolinearity? I know that multcolinearity impacts coefficients by decreasing unique variance that can be attributed to the correlated predictor variables, but I'm still not seeing why the increase in correlation increases standard errors for those particular coefficients but not others--perhaps I'm over thinking it...Is it because it is harder to discern the impact it has on other predictor variables?


Here is the gist. Lets assume some sort of optimizer is used to estimate the parameters of a linear model, and lets assume that the error function is minimized.

If there is not a lot of multi-colinearity, then the error function has a clear minimum. The optimizer is certain where to go. See the picture below. (This does not mean that the error is low.)

Clear minimum

Now take two variables with multi-colinearity; see the picture below. There is a big field (the white spot) where the error function does not change, because increasing one coefficient can be compensated with changing the other. There is a lot of uncertainty about the exact value of the parameter; hence the bigger error.


I took the pictures from SAS community: http://www.sascommunity.org/planet/blog/category/data-mining/

  • $\begingroup$ Think I'm kind of catching on...what is considered an optimizer here? And you say it is certain where it goes? So, are you saying that when they correlate that it is not certain where to go? When you say where and go, do you mean that there is no possibility that it will go beyond its limits but with correlation it is not sure where the boundaries are? ...So, with correlation error boundaries are increased?...due to, maybe, error correlating? $\endgroup$ – Axelbrain Dec 7 '15 at 21:57
  • $\begingroup$ If a gradient descent optimizer or a variation is used to estimate the parameters, then an optimizer is used to find the parameters that minimize the error function (or maximize the likelihood). The error function in the first picture has a clear minimum: the optimizer will seek that minimum out and find the correct parameters. In the second picture in the light area, the error function is more or less constant: the optimizer has a harder time to find the exact minimum. It is sliding over the white area if you will because the value of the error function is similar over a large area. $\endgroup$ – spdrnl Dec 7 '15 at 22:02
  • $\begingroup$ Thanks for the quick reply. I'll let that sink in for a bit. $\endgroup$ – Axelbrain Dec 7 '15 at 22:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.