Say we have a study where we want to run a logistic regression on a group of people, and we want to find out whether one attribute of a person makes them more likely to be a smoker.

So we have smoker yes/no as the dependent variable and let's say we have weight and gender as independent variables.

From this information, I'm guessing weight and gender are introduce collinearity because men generally weigh more than women, and therefore are not independent variables. Is this correct?

If so, I'm confused because I thought a regression controls for the other variables, so it seeks to look if men are more likely to smoke, regardless of their weight - and whether overweight people are more likely to smoke, regardless of gender.

Or do I have it all wrong?


When it comes to collinearity, we have to distinguish exact collinearity from approximate collinearity.

The former occurs when one independent variable (IV) is an exact linear combination of other independent variables. Usually this is due to some sort of elementary mistake (like entering weight in pounds and also weight in kilograms) and is easily corrected.

The latter occurs when one IV is almost a linear combination of other IVs. Assessing when "almost" is close enough to be troublesome can be done through various means, the one I think best is condition indexes and proportion of variance explained; this has been discussed elsewhere and I won't add more here.

In your case, it is very unlikely that the relationship between sex and weight is close enough to cause trouble. In a general population, it's true that men, on average, weigh more than women. But there's lots of overlap: There are plenty of light men and heavy women. If (for some reason) your sample consisted of jockeys and football players, then it would cause problems.

The problems that collinearity causes include high variances for the parameter estimates and extreme sensitivity to changes in the data. In other words, we only have a very poor idea of what the parameter estimates are and a small change in the input data can result in a huge change in the parameter estimates.


The good news is that multicollinarity has absolutely no effect on the estimated coefficients $\beta$ of the model. The bad new is that they inflate its standard error.

The formula for the standard error of a coefficient $\beta_i$ for the independent variable $X_i$ is $$ s.e.(\beta_i)=\sqrt{VIF_{X_i}\frac{\sigma_{\varepsilon}^2}{n\sigma_{X_i}^2}}$$ with $\sigma_{\varepsilon}$ being the standared error of regression (the smaller this value the larger the $R^2$ (goodness of fit) of the model), $VIF_{X_i}$ is a measure of multicolinarity (1 means no multicolinarity and the larger the more multicolinearity), $n$ is just the sample size and $\sigma_{X_i}^2$ is the variance of $X_i$.

The key point is that if you add a new variable to the model, it increases $VIF_{X_i}$ and decreases $\sigma_{\varepsilon}$. If the variable matters and is not highly correlated with $X_i$, typically the standard error $s.e.(\beta_i)$ goes down rather than up. But it is true, it can also go the other way around.

  • 2
    $\begingroup$ Perfect multicollinearity means the matrix $(X^T X)$ will not be invertible, so the a OLS solution doesn't exist. $\endgroup$ – Vimal Dec 18 '18 at 14:47
  • $\begingroup$ That is indeed correct, because in this case $VIF_{X_i}$, which is the result of regressing $X_i$ on all other dependent variables, would be undefined. However, perfect multicolinarity is in practice very rare except when working with constructed variables like Value, Cost and Value+Cost all in one model. The far more interesting case in partial multicolinarity. $\endgroup$ – Tom Pape Dec 18 '18 at 15:13

This may be too layme ...

In layman's terms "too many cooks (input series) can spoil the broth"

  • 2
    $\begingroup$ Wouldn't that be overfitting, not multicollinearity? $\endgroup$ – Paze Dec 18 '18 at 15:27
  • $\begingroup$ That's a possibility .. $\endgroup$ – IrishStat Dec 18 '18 at 16:27
  • $\begingroup$ when you bring a second series that is nearly the equivalent of the first series then that is both over-fitting and causing multicollinearity. I don't know how to define over-fitting other than the new series doesn't add anything ( a second cook for example perhaps a student of the first cook ) when 1 cook was enough . $\endgroup$ – IrishStat Dec 18 '18 at 20:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.