Multicollinearity in Multiple Regression with SPSS

Question

I want to run a multiple regression in SPSS with 7 independent variables but 3 of them are showing high correlation coefficients in the correlation matrix. How do I diagnose multicollinearity?

Answer 1

It depends on what you mean by "high." It's totally fine for different independent variables in a model to be correlated, even strongly correlated. In fact, this is the whole reason we run regression models in the first place: to look at the effect of one variable after controlling for its correlation with another.

You only run into a problem when the correlation between variable A and B is SO high that the entire idea of looking at the effect of A "holding B constant" doesn't make any sense, and the entire mathematical process breaks down (usually leading to inflated standard errors and sometimes nonsensical predictions).

You can diagnose some multicolinearity just by looking at what the variables are measuring. For example in an analysis of a particular type of workers in a unionized company, it may not make sense to look at the effect of length of employment "controlling for" wage, because due to union rules your length of employment is what determines your wage, so there are either no or very few examples of people with the same length of employment and different wages.

But you can also diagnose multicollinearity using diagnostic tests. As another post indicates, the classical way is to calculate the Variance Inflation Factor (VIF). SPSS should be able to do this as a post hoc command after the model, and it should give you a VIF value for every independent variable in the model. There is no hard and fast rule but any variable that has a VIF of over 5 (or definitely if it's over 10) has high multicollinearity with some other variable and should probably be dropped.

Answer 2

The typical way to do this is with the variance inflation factor.

The coefficient standard errors depend on four factors.

1. Sample size
2. Residual variance
3. Variance of the feature
4. Variance inflation factor

The interpretation of the variance inflation factor is that it inflates the standard error of a coefficient (so the square root of the variance of the sampling distribution) beyond where it would be for independent features. The other three factors will always be present, but if there is no multicollinearity, then the variance inflation factor will be $1$: no variance inflation.

Variance inflation factor for feature $j$ is calculated by running a linear regression that uses every other feature to predict that feature $j$ of interest. Then calculate the $R^2$ of this regression, sometimes denoted as $R^2_j$. The variance inflation factor is then $\frac{1}{1-R^2_j}$. A higher degree of feature predictiveness of this feature of interest means a higher $R^2_j$ and a higher variance inflation factor.

By calculating pairwise correlations between features, you are getting a lower bound on the variance inflation factor, since additional features cannot lower the $R^2_j$. That is, if you have correlation $r$ between two features, you know the variance inflation factor for each feature to be at least as high as $\frac{1}{1-r^2}$.

Some people give variance inflation factors of $5$ or $10$ as the thresholds to regard multicollinearity as problematic. This might serve as a decent rule of thumb, but it does hide a lot of nuance about why multicollinearity might be a problem. For instance, if the variance is inflated by a factor of $10$, but the sample size is so huge that standard errors are still going to be minuscule, this seemingly high variance inflation factor might not be such a big problem. Further, if what ultimately want to do is predict the outcome accurately instead of draw inferences about particular regression parameters, the inflation of standard errors is much less important. Validated predictive performance is what would matter most.

Note that this applies to OLS linear regression only. If you move to other models and estimation techniques (e.g., logistic regression and other GLMs, ridge regression), the techniques will differ. Generalized linear models do have generalized variance inflation factors, and the usual variance inflation factor lacks the literal interpretation that it has for OLS linear regression.