How to deal with high correlation among predictors in multiple regression?

I found a reference in an article that goes like:

According to Tabachnick & Fidell (1996) the independent variables with a bivariate correlation more than .70 should not be included in multiple regression analysis.

Problem: I used in a multiple regression design 3 variables correlated >.80, VIF's at about .2 - .3, Tolerance ~ 4- 5. I cannot exclude any of them (important predictors and outcome). When I regressed the outcome on the 2 predictors which correlated at .80, they remained both significant, each predicted important variances, and these same two variables have the largest part and semipartial correlation coefficients amongst all 10 variables included (5 controls).

Question: Is my model valid despite high correlations? Any references greatly welcomed!

I did not use Tabachnick and Fidell as a guideline, I found this reference in an article dealing with high collinearity amongst predictors.

So, basically, I have too few cases for the number of predictors in the model (many categorical, dummy coded control variables- age, tenure, gender, etc) - 13 variables for 72 cases. The Condition Index is ~ 29 with all the controls in and ~23 without them (5 variables).

I cannot drop any variable or use factorial analysis to combine them because theoretically they have sense on their own. It is too late to get more data. Since I am conducting the analysis in SPSS perhaps it would be best to find a syntax for ridge regression (although I haven't done this before and interpreting the results would be new to me).

If it matters, when I conducted stepwise regression, the same 2 highly correlated variables remained the single significant predictors of the outcome.

And I still do not understand if the partial correlations which are high for each of these variables matter as an explanation for why I have kept them in the model (in case ridge regression can't be performed).

Would you say the "Regression diagnostic: identifying influential data and sources of collinearity / David A. Belsley, Edwin Kuh and Roy E. Welsch, 1980" would be helpful in understanding multicollinearity? Or might other references be useful?

-
For an explicit example of this situation, see the analysis of 10 IVs at stats.stackexchange.com/a/14528. Here, all the IVs are strongly correlated (around 60%). But if you excluded all of them, you wouldn't have anything left! Often it's the case that you cannot drop any of these variables. This makes the T&F recommendation untenable. – whuber Sep 27 '12 at 16:20
Indeed, there are a number of pronouncements in Tabachnick and Fidell that I'd regard as at least somewhat dubious ... just because something is printed in a book doesn't mean it always makes sense. – Glen_b Feb 26 '15 at 23:43

The key problem is not correlation but collinearity (see works by Belsley, for instance). This is best tested using condition indexes (available in R, SAS and probably other programs as well. Correlation is neither a necessary nor a sufficient condition for collinearity. Condition indexes over 10 (per Belsley) indicate moderate collinearity, over 30 severe, but it also depends on which variables are involved in the collinearity.

If you do find high collinearity, it means that your parameter estimates are unstable. That is, small changes (sometimes in the 4th significant figure) in your data can cause big changes in your parameter estimates (sometimes even reversing their sign). This is a bad thing.

Remedies are 1) Getting more data 2) Dropping one variable 3) Combining the variables (e.g. with partial least squares) and 4) Performing ridge regression, which gives biased results but reduces the variance on the estimates.

-
Tabachnick and Fidell wrote a nice multivariate book for social science. They are not statististicians but their knowledge of multivariate is preety good. But I think they may create rules of thumb to simplify and could miss statistical subtleties. So I would rely more on what Peter says in his answers than in thier paper. – Michael Chernick Sep 27 '12 at 10:54
Thanks @MichaelChernick . I actually wrote my dissertation on collinearity diagnostics for multiple regression. – Peter Flom Sep 27 '12 at 10:56
I assume that you are as old as me and therefore your work came after the work of Belsley, Kuh and Welsch and Cook. I know Cook's work was mostly on other diagnostic issues (leverage and non-normality), but did he do anything on multicollinearity? Of course the concept of ridge regression even goes back before my time – Michael Chernick Sep 27 '12 at 13:21
Peter has given you even more reason to trust his answer. I did not know he was an expert in multicollinearity. – Michael Chernick Sep 27 '12 at 13:22
Hi @MichaelChernick. I am 53. Belsley wrote "the book" on collinearity. [amazon.com/…, Kuh and Welsch) covers collinearity but it is an earlier book by just [amazon.com/… that I used for my dissertation, which I started working on in 1996 or so. I don't know if Cook did anything on colinerity or not – Peter Flom Sep 27 '12 at 13:27

protected by whuber♦Feb 25 '15 at 20:33

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).