26
$\begingroup$

I found a reference in an article that goes like:

According to Tabachnick & Fidell (1996) the independent variables with a bivariate correlation more than .70 should not be included in multiple regression analysis.

Problem: I used in a multiple regression design 3 variables correlated >.80, VIF's at about .2 - .3, Tolerance ~ 4- 5. I cannot exclude any of them (important predictors and outcome). When I regressed the outcome on the 2 predictors which correlated at .80, they remained both significant, each predicted important variances, and these same two variables have the largest part and semipartial correlation coefficients amongst all 10 variables included (5 controls).

Question: Is my model valid despite high correlations? Any references greatly welcomed!

Thank you for the answers!

I did not use Tabachnick and Fidell as a guideline, I found this reference in an article dealing with high collinearity amongst predictors.

So, basically, I have too few cases for the number of predictors in the model (many categorical, dummy coded control variables- age, tenure, gender, etc) - 13 variables for 72 cases. The Condition Index is ~ 29 with all the controls in and ~23 without them (5 variables).

I cannot drop any variable or use factorial analysis to combine them because theoretically they have sense on their own. It is too late to get more data. Since I am conducting the analysis in SPSS perhaps it would be best to find a syntax for ridge regression (although I haven't done this before and interpreting the results would be new to me).

If it matters, when I conducted stepwise regression, the same 2 highly correlated variables remained the single significant predictors of the outcome.

And I still do not understand if the partial correlations which are high for each of these variables matter as an explanation for why I have kept them in the model (in case ridge regression can't be performed).

Would you say the "Regression diagnostic: identifying influential data and sources of collinearity / David A. Belsley, Edwin Kuh and Roy E. Welsch, 1980" would be helpful in understanding multicollinearity? Or might other references be useful?

Improve this question
$\endgroup$
2
  • 3
    $\begingroup$ For an explicit example of this situation, see the analysis of 10 IVs at stats.stackexchange.com/a/14528. Here, all the IVs are strongly correlated (around 60%). But if you excluded all of them, you wouldn't have anything left! Often it's the case that you cannot drop any of these variables. This makes the T&F recommendation untenable. $\endgroup$
    – whuber ♦
    Commented Sep 27, 2012 at 16:20
  • 1
    $\begingroup$ Indeed, there are a number of pronouncements in Tabachnick and Fidell that I'd regard as at least somewhat dubious ... just because something is printed in a book doesn't mean it always makes sense. $\endgroup$
    – Glen_b
    Commented Feb 26, 2015 at 23:43

1 Answer 1

Reset to default
30
$\begingroup$

The key problem is not correlation but collinearity (see works by Belsley, for instance). This is best tested using condition indexes (available in R, SAS and probably other programs as well. Correlation is neither a necessary nor a sufficient condition for collinearity. Condition indexes over 10 (per Belsley) indicate moderate collinearity, over 30 severe, but it also depends on which variables are involved in the collinearity.

If you do find high collinearity, it means that your parameter estimates are unstable. That is, small changes (sometimes in the 4th significant figure) in your data can cause big changes in your parameter estimates (sometimes even reversing their sign). This is a bad thing.

Remedies are

  1. Getting more data
  2. Dropping one variable
  3. Combining the variables (e.g. with partial least squares) and
  4. Performing ridge regression, which gives biased results but reduces the variance on the estimates.
Improve this answer
$\endgroup$
13
  • 1
    $\begingroup$ Tabachnick and Fidell wrote a nice multivariate book for social science. They are not statististicians but their knowledge of multivariate is preety good. But I think they may create rules of thumb to simplify and could miss statistical subtleties. So I would rely more on what Peter says in his answers than in thier paper. $\endgroup$
    – Michael R. Chernick
    Commented Sep 27, 2012 at 10:54
  • $\begingroup$ Thanks @MichaelChernick . I actually wrote my dissertation on collinearity diagnostics for multiple regression. $\endgroup$
    – Peter Flom
    Commented Sep 27, 2012 at 10:56
  • $\begingroup$ I assume that you are as old as me and therefore your work came after the work of Belsley, Kuh and Welsch and Cook. I know Cook's work was mostly on other diagnostic issues (leverage and non-normality), but did he do anything on multicollinearity? Of course the concept of ridge regression even goes back before my time $\endgroup$
    – Michael R. Chernick
    Commented Sep 27, 2012 at 13:21
  • 1
    $\begingroup$ @Peter Flom: Why is correlation neither a necessary nor a sufficient condition for collinearity? Are you referring to non-linear correlation? $\endgroup$
    – Funkwecker
    Commented Oct 14, 2016 at 7:27
  • 7
    $\begingroup$ It's not necessary because, if there are a large number of variables, all pairs can be only slightly correlated yet the sum of them is perfectly colinear. It's not sufficient because there are cases where fairly high correlation does not yield troublesome collinearity per condition indexes $\endgroup$
    – Peter Flom
    Commented Oct 14, 2016 at 11:20

Not the answer you're looking for? Browse other questions tagged or ask your own question.