11
$\begingroup$

I want to detect whether collinearity is a problem in my OLS regression. I understand that variance inflation factors and the condition index are two commonly used measures, but am finding it difficult to find anything definite on the merits of each approach, or what the scores should be.

A prominent source which indicates what approach to do, and/or what scores are appropriate would be very useful.

A similar question was asked at "Is there a reason to prefer a specific measure of multicollinearity?" but I'm ideally after a reference that I can cite .

$\endgroup$
  • 4
    $\begingroup$ Do remember that collinearity is very much a matter of degree, so even if you find a text that gives a nice, citable number it's not something you should treat as a hard cut-off value for "no problem" vs "we have a problem". $\endgroup$ – Silverfish Sep 22 '15 at 21:03
  • 4
    $\begingroup$ @Silverfish gives good advice. Belsley, Kuh and Welsch also emphasize that collinearity, even when present, is not necessarily harmful: you have to determine whether it's actually causing a problem for your analysis. $\endgroup$ – whuber Sep 22 '15 at 21:39
11
$\begingroup$

Belsley, Kuh, and Welsch is the text to go to for this kind of question. They include extensive discussion of older diagnostics in a section entitled "Historical Perspective". Concerning VIF they write

... If we assume the $X$ data have been centered and scaled to have unit length, the correlation matrix $R$ is simply $X^\prime X$. ...

We are considering $R^{-1} = (X^\prime X)^{-1}$. The diagonal elements of $R^{-1}$, the $r^{ii}$, are often called the variance inflation factors, $\text{VIF}_i$, and their diagnostic value follows from the relation $$\text{VIF}_i = \frac{1}{1 - R_i^2}$$ where $R_i^2$ is the multiple correlation coefficient of $X_i$ regressed on the remaining explanatory variables. Clearly a high VIF indicates an $R_i^2$ near unity, and hence points to collinearity. This measure is therefore of some use as an overall indication of collinearity. Its weaknesses, like those of $R$, lie in its inability to distinguish among several coexisting near dependencies and in the lack of a meaningful boundary to distinguish between values of VIF that can be considered high and those that can be considered low.

In place of analyzing $R$ (or $R^{-1}$), BKW propose careful, controlled examination of the Singular Value Decomposition of $X$. They motivate it by demonstrating that the ratio of the largest to the smallest singular values is the condition number of $X$ and show how the condition number provides (at times tight) bounds on the the propagation of computing errors in the calculation of the regression estimates. They go on to attempt an approximate decomposition of the variances of the parameter estimates $\hat\beta_i$ into components associated with the singular values. The power of this decomposition lies in its ability (in many cases) to reveal the nature of the collinearity, rather than just indicating its presence.

Anyone who has built regression models with hundreds of variables will appreciate this feature! It is one thing for the software to say "your data are collinear, I cannot proceed" or even to say "your data are collinear, I'm throwing out the following variables." It is altogether a much more useful thing for it to be able to say "the group of variables $X_{i_1}, \ldots, X_{i_k}$ is causing instabilities in the calculations: see which of those variables you can do without or consider performing a principal components analysis to reduce their number."

Ultimately, BKW recommend diagnosing collinearity by means of

... the following double condition:

  1. A singular value judged to have a high condition index, and which is associated with
  2. High variance-decomposition proportions for two or more estimated regression coefficient variances.

The number of condition indexes deemed large (say, greater than $30$) in (1) identifies the number of near dependencies among the columns of the data matrix $X$, and the magnitudes of these high condition indexes provide a measure of their relative "tightness." Furthermore, the determination in (2) of large variance-decomposition proportions (say, greater than $0.5$) associated with each high condition index identifies those variates that are involved in the corresponding near dependency, and the magnitude of these proportions in conjunction with the high condition index provides a measure of the degree to which the corresponding regression estimate has been degraded by the presence of collinearity.

| cite | improve this answer | |
$\endgroup$
10
$\begingroup$
  • Variance Inflation Factors (VIFs) are easy to understand. Regress each of your design matrix columns on all the others, note the $R^2$ of this model, calculate $1/(1-R^2)$, and there you go. A VIF of 10 means that you can explain 90% of one predictor's variance using all the other regressors. This is typically used as a rule of thumb threshold for collinearity.

    However, VIFs as typically implemented cannot tell you about collinearity with the intercept, since the intercept is usually silently included in these "helper" regressions. In addition, if a regressor has a high VIF, you don't immediately know which other regressors are responsible for the collinearity. You'd need to look at standardized coefficients in the helper regressions.

  • The condition indices and collinearity decomposition proportions of Belsley, Kuh & Welsch (Belsley, D. A.; Kuh, E. & Welsch, R. E. Regression Diagnostics: Identifying Influential Data and Sources of Collinearity. John Wiley & Sons, 1980) are far harder to understand. I used to work with these a few years back, but I won't attempt to explain them here without getting a refresher ;-)

    These diagnostics do allow detecting collinearity with the intercept. And you can investigate the collinearity decomposition proportions to deduce which other regressors are responsible for one given regressor's collinearity.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks - very useful - are you by chance aware of a cite for the VIF greater than 10 rule-of-thumb ... I can find in plenty of econ-lecture notes, but can't find anything published that says that... $\endgroup$ – kyrenia Sep 22 '15 at 15:43
  • $\begingroup$ @kyrenia "greater than 10" is far from the only threshold I have seen suggested! I wonder whether there is variation between fields, or simply between authors. $\endgroup$ – Silverfish Sep 22 '15 at 20:56
  • 3
    $\begingroup$ @Silverfish Definitely there is variation between fields. I have been told of experimental design people teaching that a VIF over $2$ needs to be corrected! For observational studies, there also should be variation depending on the number of regressors: the more there are, the larger the VIFs will get just due to chance fluctuation. With hundreds of variables in many cases you will just have to put up with VIFs of $100$ or larger. Presumably there's also enough data in such cases to make up for such variance inflation. $\endgroup$ – whuber Sep 22 '15 at 23:31
  • $\begingroup$ @whuber Thanks for that. That is a very interesting observation and highly relevant to the question posed by the OP: given the "second rank" importance of comments within the StackExchange system, I think you should consider incorporating that into your excellent answer. $\endgroup$ – Silverfish Sep 22 '15 at 23:37
6
$\begingroup$

For widely available references to cite, the Faraway book on page 117 provides a rule of thumb of above 30 for detecting problems based on condition numbers, and An Introduction to Statistical Learning, page 101, says that VIF values above 5 or 10 indicate a problem.

Probably more important than which method you use for identifying multicollinearity will be how you deal with it.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.