I have fitted two different binomial logistic regression models A and B. Model A contains only one predictor variable. Model B contains a different set of predictor variables, none of which is the predictor included in model A. There is a notable degree of multicollinearity between the predictors in model B. I want to compare how well the two models can account for the variation in my data.

Usually, when discussing the consequences of multicollinearity in linear regression models, most authors focus on the effect on the predictors. For example Dormann et al (2012) point out that the standard errors of collinear coefficients will be inflated, which leads to "inaccurate tests of significance for the predictors, meaning that important predictors may not be significant, even if they are truly influential" (p. 29).

However, what is less clear to me is the effect of multicollinearity on the overall performance of the model. This question asks whether multicollinearity affects the performance of the model as a classifier, with reference to the Wikipedia article on Multicollinearity which says that "multicollinearity does not reduce the predictive power or reliability of the model as a whole, at least within the sample data set". In his answer to the question, @EdM seems to confirm that multicollinearity does not affect model reliability unless it is used to predict a data set different from the one used to fit it.

My case is somewhat different because I don't want to use the models as classifiers on new data. Instead, I want to compare how well they can explain my data set. So, that answer still leaves me with the following questions:

  • Is it valid to say that the explained variance of a model is invariant to the presence of multicollinearity?
  • Can I use still measures such as AIC or AUROC to compare the performance of my models A and B even though the predictors of model B are strongly correlated?
  • Is there a quotable reference that discusses the effect of multicollinearity on the explained variance of models and on measures such as the AIC or AUROC?
  • $\begingroup$ What a dynamite question. btw I'll say Yes to the first 2 questions. I am curious, though, why within-sample performance is all you care about here. $\endgroup$
    – rolando2
    Feb 3, 2017 at 18:42
  • $\begingroup$ @rolando2: There's a claim made by one author that his novel predictor is as good as the established predictors discussed by other authors. My data set comes from an experiment specifically designed to test the effect of the novel predictor in comparison to the combined effect of the established predictors. The point of the analysis is not to tease apart the effect of the established predictors, or to verify that they actually work, the point is only to see whether the model using the novel predictor performs as good as the model using the established predictors. $\endgroup$
    – Schmuddi
    Feb 3, 2017 at 18:51
  • $\begingroup$ Thanks....A predictor requires a coefficient :-) How will anyone know that the within-sample results give the best, most reliable estimates of this novel predictor's coefficient...standardized coefficient...standard error...or, generally, performance? $\endgroup$
    – rolando2
    Feb 3, 2017 at 19:18
  • 1
    $\begingroup$ @Schmuddi You mean a reviewer said that? It could be helpful to quote his or her comment exactly so we can help you decide on a reply. $\endgroup$ Feb 4, 2017 at 0:15
  • 1
    $\begingroup$ As shown at stats.stackexchange.com/questions/447684, in any model that is fit by optimizing some function of a linear combination of the variables, it doesn't matter how you express the variables: all that matters is the vector space they generate. That instantly answers your question: multicollinearity is irrelevant. What might be relevant is the possibility that future data could vary so much from the data you have that your predictions become gross extrapolations. stats.stackexchange.com/a/507178/919 is also relevant. $\endgroup$
    – whuber
    May 3 at 17:02

1 Answer 1


From the comments, it is evident that you mean some kind of predictive accuracy measurement, any of which would be a function of the predictions and the true observations.

Multicollinearity cannot affect the true observations, since they are the observations even if you don't do any modeling, so for multicollinearity to influence the predictive accuracy, multicollinearity must influence the predictions.

Let's do a simulation where we do a logistic regression on two correlated variables, regress on a transformation of those variables that retains all of the information while removing the correlation, and comparing predictions.

N <- 25
X <- MASS::mvrnorm(N, c(0, 0), matrix(c(1, 0.9, 0.9, 1), 2, 2))
y <- rbinom(N, 1, 0.5)
L_correlated <- glm(y ~ X, family = binomial)
predictions_correlated <- predict(L_correlated)
PCs <- X %*% prcomp(X)$rotation # Do PCA to remove correlation
L_uncorrelated <- glm(y ~ PCs, family = binomial) # Take all PCs to
                                                  # retain all information
predictions_uncorrelated <- predict(L_uncorrelated)
round(predictions_correlated - predictions_uncorrelated, 14)

Aside from floating point arithmetic issues out past the $14^{\text{th}}$ decimal place, the predictions are the same.

In GLMs, multicollinearity can affect interpretations, p-values, and confidence intervals, but multicollinearity does not affect the predicted values. Some people also dislike multicollinearity because they think they can drop some variables and retain most of the information while lowering the parameter count. As much as that makes sense, it is not perfect and might even be fairly problematic.

  • 2
    $\begingroup$ I have to admit that by now, I completely forgot about my question, and I had to re-read everything to understand my own problem back then. But this is exactly the type of argument that I could've needed back then – thanks! $\endgroup$
    – Schmuddi
    May 3 at 17:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.