Multicollinearity when individual regressions are significant, but VIFs are low

Question

I have 6 variables ($x_{1}...x_{6}$) that I am using to predict $y$. When performing my data analysis, I first tried a multiple linear regression. From this, only two variables were significant. However, when I ran a linear regression comparing each variable individually to $y$, all but one were significant ($p$ anywhere from less than 0.01 to less than 0.001). It was suggested that this was due to multicollinearity.

My initial research on this suggests checking for multicollinearity by using VIFs. I downloaded the appropriate package from R, and ended up with the resulting VIFs: 3.35, 3.59, 2.64, 2.24, and 5.56. According to various sources online, the point you should be worried about multicollinearity with your VIFs is either at 4 or 5.

I am now stumped about what this means for my data. Do I or do I not have a multicollinearity problem? If I do, then how should I proceed? (I cannot collect more data, and the variables are parts of a model that aren't obviously related) If I don't have this problem, then what should I be taking from my data, particularly the fact that these variables are highly significant individually, but not significant at all when combined.

Edit: Some questions have been asked regarding the dataset, and so I would like to expand...

In this particular case, we're looking to understand how specific social cues (gesture, gaze, etc) affect the likelihood of someone producing some other cue. We would like our model to include all significant attributes, so I am uncomfortable removing some that seem redundant.

There are not any hypotheses with this right now. Rather, the problem is unstudied, and we are looking to gain a better understanding of what attributes are important. As far as I can tell, these attributes should be relatively independent of one another (you can't just say gaze and gestures are the same, or one the subset of another). It would be nice to be able to report p values for everything, since we would like other researchers to understand what has been looked at.

Edit 2: Since it came up somewhere below, my $n$ is 24.

Answer 1

To understand what can go on, it is instructive to generate (and analyze) data that behave in the manner described.

For simplicity, let's forget about that sixth independent variable. So, the question describes regressions of one dependent variable $y$ against five independent variables $x_1, x_2, x_3, x_4, x_5$, in which

• Each ordinary regression $y \sim x_i$ is significant at levels from $0.01$ to less than $0.001$.

• The multiple regression $y \sim x_1 + \cdots + x_5$ yields significant coefficients only for $x_1$ and $x_2$.

• All variance inflation factors (VIFs) are low, indicating good conditioning in the design matrix (that is, lack of collinearity among the $x_i$).

Let's make this happen as follows:

1. Generate $n$ normally distributed values for $x_1$ and $x_2$. (We will choose $n$ later.)

2. Let $y = x_1 + x_2 + \varepsilon$ where $\varepsilon$ is independent normal error of mean $0$. Some trial and error is needed to find a suitable standard deviation for $\varepsilon$; $1/100$ works fine (and is rather dramatic: $y$ is extremely well correlated with $x_1$ and $x_2$, even though it is only moderately correlated with $x_1$ and $x_2$ individually).

3. Let $x_j$ = $x_1/5 + \delta$, $j=3,4,5$, where $\delta$ is independent standard normal error. This makes $x_3,x_4,x_5$ only slightly dependent on $x_1$. However, via the tight correlation between $x_1$ and $y$, this induces a tiny correlation between $y$ and these $x_j$.

Here's the rub: if we make $n$ large enough, these slight correlations will result in significant coefficients, even though $y$ is almost entirely "explained" by only the first two variables.

I found that $n=500$ works just fine for reproducing the reported p-values. Here's a scatterplot matrix of all six variables:

By inspecting the right column (or the bottom row) you can see that $y$ has a good (positive) correlation with $x_1$ and $x_2$ but little apparent correlation with the other variables. By inspecting the rest of this matrix, you can see that the independent variables $x_1, \ldots, x_5$ appear to be mutually uncorrelated (the random $\delta$ mask the tiny dependencies we know are there.) There are no exceptional data--nothing terribly outlying or with high leverage. The histograms show that all six variables are approximately normally distributed, by the way: these data are as ordinary and "plain vanilla" as one could possibly want.

In the regression of $y$ against $x_1$ and $x_2$, the p-values are essentially 0. In the individual regressions of $y$ against $x_3$, then $y$ against $x_4$, and $y$ against $x_5$, the p-values are 0.0024, 0.0083, and 0.00064, respectively: that is, they are "highly significant." But in the full multiple regression, the corresponding p-values inflate to .46, .36, and .52, respectively: not significant at all. The reason for this is that once $y$ has been regressed against $x_1$ and $x_2$, the only stuff left to "explain" is the tiny amount of error in the residuals, which will approximate $\varepsilon$, and this error is almost completely unrelated to the remaining $x_i$. ("Almost" is correct: there is a really tiny relationship induced from the fact that the residuals were computed in part from the values of $x_1$ and $x_2$ and the $x_i$, $i=3,4,5$, do have some weak relationship to $x_1$ and $x_2$. This residual relationship is practically undetectable, though, as we saw.)

The conditioning number of the design matrix is only 2.17: that's very low, showing no indication of high multicollinearity whatsoever. (Perfect lack of collinearity would be reflected in a conditioning number of 1, but in practice this is seen only with artificial data and designed experiments. Conditioning numbers in the range 1-6 (or even higher, with more variables) are unremarkable.) This completes the simulation: it has successfully reproduced every aspect of the problem.

The important insights this analysis offers include

1. p-values don't tell us anything directly about collinearity. They depend strongly on the amount of data.

2. Relationships among p-values in multiple regressions and p-values in related regressions (involving subsets of the independent variable) are complex and usually unpredictable.

Consequently, as others have argued, p-values should not be your sole guide (or even your principal guide) to model selection.

---

Edit

It is not necessary for $n$ to be as large as $500$ for these phenomena to appear. Inspired by additional information in the question, the following is a dataset constructed in a similar fashion with $n=24$ (in this case $x_j = 0.4 x_1 + 0.4 x_2 + \delta$ for $j=3,4,5$). This creates correlations of 0.38 to 0.73 between $x_{1-2}$ and $x_{3-5}$. The condition number of the design matrix is 9.05: a little high, but not terrible. (Some rules of thumb say that condition numbers as high as 10 are ok.) The p-values of the individual regressions against $x_3, x_4, x_5$ are 0.002, 0.015, and 0.008: significant to highly significant. Thus, some multicollinearity is involved, but it's not so large that one would work to change it. The basic insight remains the same: significance and multicollinearity are different things; only mild mathematical constraints hold among them; and it is possible for the inclusion or exclusion of even a single variable to have profound effects on all p-values even without severe multicollinearity being an issue.

x1 x2 x3 x4 x5 y
-1.78256    -0.334959   -1.22672    -1.11643    0.233048    -2.12772
0.796957    -0.282075   1.11182 0.773499    0.954179    0.511363
0.956733    0.925203    1.65832 0.25006 -0.273526   1.89336
0.346049    0.0111112   1.57815 0.767076    1.48114 0.365872
-0.73198    -1.56574    -1.06783    -0.914841   -1.68338    -2.30272
0.221718    -0.175337   -0.0922871  1.25869 -1.05304    0.0268453
1.71033 0.0487565   -0.435238   -0.239226   1.08944 1.76248
0.936259    1.00507 1.56755 0.715845    1.50658 1.93177
-0.664651   0.531793    -0.150516   -0.577719   2.57178 -0.121927
-0.0847412  -1.14022    0.577469    0.694189    -1.02427    -1.2199
-1.30773    1.40016 -1.5949 0.506035    0.539175    0.0955259
-0.55336    1.93245 1.34462 1.15979 2.25317 1.38259
1.6934  0.192212    0.965777    0.283766    3.63855 1.86975
-0.715726   0.259011    -0.674307   0.864498    0.504759    -0.478025
-0.800315   -0.655506   0.0899015   -2.19869    -0.941662   -1.46332
-0.169604   -1.08992    -1.80457    -0.350718   0.818985    -1.2727
0.365721    1.10428 0.33128 -0.0163167  0.295945    1.48115
0.215779    2.233   0.33428 1.07424 0.815481    2.4511
1.07042 0.0490205   -0.195314   0.101451    -0.721812   1.11711
-0.478905   -0.438893   -1.54429    0.798461    -0.774219   -0.90456
1.2487  1.03267 0.958559    1.26925 1.31709 2.26846
-0.124634   -0.616711   0.334179    0.404281    0.531215    -0.747697
-1.82317    1.11467 0.407822    -0.937689   -1.90806    -0.723693
-1.34046    1.16957 0.271146    1.71505 0.910682    -0.176185

Answer 2

Do I or do I not have a multicollinearity problem? If I do, then how should I proceed?

It's not an either-or situation. And I am skeptical about the "4 or 5" guideline. For each of your predictors, the standard error of the coefficient is between 2.2 and 5.6 times as large as it would be if the predictor were uncorrelated with the others. And the portion of a given predictor that cannot be explained by the others ranges from 1/2.2 to 1/5.6, or 18% to 45%. Altogether, that seems a pretty substantial amount of collinearity.

But let's step back for a minute. Are you really trying to predict *Y*, as opposed to trying to explain it? If the former, then I don't suppose you need care whether the significance level of a given variable changes when others are present in the model. Your job is really much easier than it would be if true explanation were needed.

If explanation is your goal, you'll need to consider the way these variables interrelate--something that requires more than statistical information. Clearly they overlap in the way they relate to Y, and this collinearity will make it difficult to establish, for example, their rank order of importance in accounting for Y. In this situation there's no one clear path for you to follow.

In any case, I hope you are considering methods of crossvalidation.

Answer 3

You have multicollinearity. Your initial analysis demonstrated that. As far as it being a problem, that's another question that seems to have many answers in your case.

Maybe if you got the basic issue better it would be more obvious what to do?...

With the multicollinearity your regression coefficients are about the unique (well closer to unique) contributions of each variable to your model. If some are correlated with each other then each correlated one's unique contribution is smaller. That's probably partially why none are significant when they're all there together but when used alone they can be.

The first thing you likely need to do is consider what the intercorrelation among your variables means. For example, do you have a bunch of variables that just stand for the same thing? Did you just happen to measure your predictors over a poor scale and get incidental correlations? Don't try to fix the regression, try to understand your variables.

Consider X1 and X2 with a very strong correlation between them, say r = 0.90. If you put X1 in the model and it's a significant predictor then another model with X2 alone will very likely be significant as well because they're almost the same thing. If you put them in the model together at least one of them has to suffer because the multiple regression is going to solve to their unique contributions. They might both be non-significant. But that's not the point, the point is recognizing why they overlap so much and if they even say anything different from one another and whether you need them or not? Maybe one expresses an idea more meaningfully and more related to your response variable than the other. Maybe you'll conclude that they're the same thing with different levels of variability.

Also, when looking at models of any kind, but especially with intercorrelated predictors, p-values are a terrible way to tell if a new predictor makes a meaningful contribution (if that's what you're trying to do... not sure what you're trying to do because it sounds like you're just trying to make the regression either A) simple, or B) come out the way you want... neither of which are feasible). You're probably best off looking at AIC to help you determine which predictors you should keep and which don't contribute anything.

Answer 4

Personally, I'd use condition indexes and the variance explained table to analyze collinearity.

I would also not use p values as a criterion for model building, and when comparing models with 6 IVs to models with 1, I'd look at changes in the effect size of the parameter for the variable that is both.

But you can certainly have the results you mention without collinearity. Collinearity is only about the X variables and their relationship. But two variables could both relate strongly to Y while not relating strongly to each other.

Answer 5

Regarding multicollinearity there are various thresholds being mentioned usually converging around a VIF of 10 corresponding to an underlying R Square value of 0.90 between the tested variable vs the other independent variables. The VIFs of your variables appear passable, and you could technically keep them in a model.

Yet, I would use a stepwise regression method to see which are the best combination of variables and how much more explanation (incremental increase in R Square) you get by adding variables. The arbitrating benchmark should be the Adjusted R Square value that adjusts the R Square value downward by penalizing the model for adding variables.

Your variables are somewhat correlated with each other. This is inevitable, it is just a matter of degree. Given the VIFs you mention, I suspect intuitively that you will get the vast majority of the information/explanation bit from the best 2 variable combination. And, that adding variables may add only marginal incremental value.

When looking at the combination of variables that are selected by the stepwise regression process, I would also look at what variables are selected and if their regression coefficient signs are consistent with their correlation with y. If they are not, it can be due to a legitmate interaction between the variables. But, it could also be a result of model overfitting and that the regression coefficients are spurious. They reflect a mathematical fit, but are meaningless in terms of underlying causality.

Another way to select your variables is to decide from a logic standpoint which ones are the main 2 or 3 variables that should be in the model. You start with those and then check how much more information do you get by adding a variable. Check the adjusted R Square, consistency of the regression coefficient relative to the original regression, and obviously test all the models with hold out period. Pretty soon, it will be evident what is your best model.

Answer 6

If your explanatory variables are count data, and it is not unreasonable to assume that they are normally distributed, you can transform them into standard normal variates using the R scale command. Doing this can reduce the collinearity. But that will probably not solve the whole problem.

A useful batch of R commands for analyzing and dealing with collinearity are found on Florian Jaeger's blog, including:

z. <- function (x) scale(x)
r. <- function (formula, ...) rstandard(lm(formula, ...))

The z. function converts a vector into a standard normal variate. The r. function returns standardized residuals for regressing one predictor against another. You can use this to effectively divide the model deviance into different tranches so that only some variables have access to the most senior tranche, then the next tranche will be offered to residualized variables. (Sorry for my homespun terminology) So if a model of the form

Y ~ A + B

suffers from multicollinearity, then you can run either of

Y ~ A + r.(B)
Y ~ r.(A) + B

so that only the residuals of the "junior tranche" variable (when regressed against the "senior tranche" variable) are fitted to the model. This way, you are shielded from multicollinearity, but have a more complicated set of parameters to report.