I have a large set of data and a couple of regressors that seem to be somewhat to highly correlated. I will include these in a GLM and am primarily interested in the predictive ability of the model and not inference on individual parameter estimates.

How will the predictive ability of the model be influenced by the multicollinearity? Might if get outlying predictions?

  • $\begingroup$ As an example, collect some measurements of the side lengths $X_1$ and $X_2$ of a set of squares, along with their perimeters $Y$. Here, the multicollinearity is perfect (because $X_1=X_2$ within measurement error) and the predictability is perfect, too. However, there are infinitely many possible formulas leading to the same prediction, such as $4X_1$, $4X_2$, $2X_1+2X_2$, $10X_1-6X_2$, etc. $\endgroup$ – whuber Sep 10 '12 at 15:59

In the context of the Normal GLM, multicollinearity isn't always such a problem for prediction. Often it can mean that although individual coefficients can't be estimated efficiently, the linear combination of them (i.e. the fitted values $X\boldsymbol{\hat{\beta}}$) can still be. This tends to be the case when tests on the model as a whole suggest it is a good fit to the data, but $t$-statistics and the like suggest that individual coefficients aren't significant.

The model can still be good for prediction provided that the covariates for the response ($y$) you are trying to predict are similarly correlated with each other as those used to fit the model.

Sections 10.8 and 10.9 of Basic Econometrics by Gujarati (2003) for more detail and guidance here, as well as some good examples for Economic data.

  • $\begingroup$ Thanks, so what you are saying is that as long as the underlying correlation structure does not change the model will still be statisfactory. You are reffering in particular to the normal distribution, why would it be different for other exponential family distributions? $\endgroup$ – Patrik Emanuelsson Sep 10 '12 at 14:32
  • $\begingroup$ Yes, that's what I'm saying. I don't imagine that things would be different for other exponential family distributions, but I'm not sure for certain so I was being overly cautious. The text book I refer to discusses these things in terms of the OLS estimator, which is equivalent to ML estimation for Normal data. The IRWLS algorithm - which is used to fit parameters for a GLM - also produces estimates which correspond to MLEs, so I know for Normal data all three approaches will give the same results. $\endgroup$ – Sam Livingstone Sep 10 '12 at 17:09
  • $\begingroup$ For other EF distributions I don't think the IRWLS algorithm gives equivalent results to simply performing OLS on a transformed response variable (such as log(y) for Poisson data), so I can't give such a precise answer. $\endgroup$ – Sam Livingstone Sep 10 '12 at 17:11

if you have multicollinearity then the high correlation could mean that two or more variables are close to being linearly related. This means that more than one set of coefficietns can describe the model. So alternative choices of coefficients could be equally good. This means the coefficient estimates are unstable. But the model accuracy is not affected. So including both variables does not hurt the model prediction necessarily. However, you may have a misleading sense that there is a gain in accuracy by including both. It is possible that prediction is not imporved.


Graphical explanation of discriminating between low and high risk:enter image description here

I think the other comments address the mathematics. What the figure shows is two highly correlated predictors, #1 with moderate discrimination and #2 with very little discrimination between the low and high risk. The correlation actually leads to better (nearly perfect) discrimination, even though the individual coefficients are poorly estimated. The discriminant function is well estimated.

  • $\begingroup$ Welcome to the site, @TimChruch. This is interesting, but it is rather sparse by our standards. Can you add some text to explain what the figure shows, how it is related to the question, & the answer that it provides? Eg, I don't quite see how it answers the explicit question, "How will the predictive ability of the model be influenced by the multicollinearity?". $\endgroup$ – gung Feb 7 '17 at 19:20
  • 1
    $\begingroup$ @Tim Church You should edit your answer and add the material from your comment. $\endgroup$ – Mark L. Stone Feb 7 '17 at 19:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.