Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have 6 variables ($x_{1}...x_{6}$) that I am using to predict $y$. When performing my data analysis, I first tried a multiple linear regression. From this, only two variables were significant. However, when I ran a linear regression comparing each variable individually to $y$, all but one were significant ($p$ anywhere from less than 0.01 to less than 0.001). It was suggested that this was due to multicollinearity.

My initial research on this suggests checking for multicollinearity by using VIFs. I downloaded the appropriate package from R, and ended up with the resulting VIFs: 3.35, 3.59, 2.64, 2.24, and 5.56. According to various sources online, the point you should be worried about multicollinearity with your VIFs is either at 4 or 5.

I am now stumped about what this means for my data. Do I or do I not have a multicollinearity problem? If I do, then how should I proceed? (I cannot collect more data, and the variables are parts of a model that aren't obviously related) If I don't have this problem, then what should I be taking from my data, particularly the fact that these variables are highly significant individually, but not significant at all when combined.

Edit: Some questions have been asked regarding the dataset, and so I would like to expand...

In this particular case, we're looking to understand how specific social cues (gesture, gaze, etc) affect the likelihood of someone producing some other cue. We would like our model to include all significant attributes, so I am uncomfortable removing some that seem redundant.

There are not any hypotheses with this right now. Rather, the problem is unstudied, and we are looking to gain a better understanding of what attributes are important. As far as I can tell, these attributes should be relatively independent of one another (you can't just say gaze and gestures are the same, or one the subset of another). It would be nice to be able to report p values for everything, since we would like other researchers to understand what has been looked at.

Edit 2: Since it came up somewhere below, my $n$ is 24.

share|improve this question
    
Assuming you do have multicollinearity, can you expand, as @rolando2 suggests, on the intended purpose of the model? Is it the case that all of the predictors are of importance to other investigators (in which case you would like to report significance levels for each of them), or could you just throw one or two of them out? –  jlovegren Mar 11 '12 at 18:06
    
@jlovegren I added some information above - let me know if you need more info. –  allie Mar 11 '12 at 22:41
    
Are the explanatory variables measured on a continuous scale? In that case, there are methods for residualizing which are not too difficult. If they are categorical, I don't know but I hope someone else would (I have asked a similar question on this site). –  jlovegren Mar 11 '12 at 23:20
    
@jlovegren Five of the six variables are counts. –  allie Mar 12 '12 at 1:31
    
one more thing, just to be sure. do the counts have a clear upper limit which is frequently achieved, or if the maximum value of the count unbounded in principle? –  jlovegren Mar 12 '12 at 2:09

6 Answers 6

up vote 12 down vote accepted

To understand what can go on, it is instructive to generate (and analyze) data that behave in the manner described.

For simplicity, let's forget about that sixth independent variable. So, the question describes regressions of one dependent variable $y$ against five independent variables $x_1, x_2, x_3, x_4, x_5$, in which

  • Each ordinary regression $y \sim x_i$ is significant at levels from $0.01$ to less than $0.001$.

  • The multiple regression $y \sim x_1 + \cdots + x_5$ yields significant coefficients only for $x_1$ and $x_2$.

  • All variance inflation factors (VIFs) are low, indicating good conditioning in the design matrix (that is, lack of collinearity among the $x_i$).

Let's make this happen as follows:

  1. Generate $n$ normally distributed values for $x_1$ and $x_2$. (We will choose $n$ later.)

  2. Let $y = x_1 + x_2 + \varepsilon$ where $\varepsilon$ is independent normal error of mean $0$. Some trial and error is needed to find a suitable standard deviation for $\varepsilon$; $1/100$ works fine (and is rather dramatic: $y$ is extremely well correlated with $x_1$ and $x_2$, even though it is only moderately correlated with $x_1$ and $x_2$ individually).

  3. Let $x_j$ = $x_1/5 + \delta$, $j=3,4,5$, where $\delta$ is independent standard normal error. This makes $x_3,x_4,x_5$ only slightly dependent on $x_1$. However, via the tight correlation between $x_1$ and $y$, this induces a tiny correlation between $y$ and these $x_j$.

Here's the rub: if we make $n$ large enough, these slight correlations will result in significant coefficients, even though $y$ is almost entirely "explained" by only the first two variables.

I found that $n=500$ works just fine for reproducing the reported p-values. Here's a scatterplot matrix of all six variables:

spm

By inspecting the right column (or the bottom row) you can see that $y$ has a good (positive) correlation with $x_1$ and $x_2$ but little apparent correlation with the other variables. By inspecting the rest of this matrix, you can see that the independent variables $x_1, \ldots, x_5$ appear to be mutually uncorrelated (the random $\delta$ mask the tiny dependencies we know are there.) There are no exceptional data--nothing terribly outlying or with high leverage. The histograms show that all six variables are approximately normally distributed, by the way: these data are as ordinary and "plain vanilla" as one could possibly want.

In the regression of $y$ against $x_1$ and $x_2$, the p-values are essentially 0. In the individual regressions of $y$ against $x_3$, then $y$ against $x_4$, and $y$ against $x_5$, the p-values are 0.0024, 0.0083, and 0.00064, respectively: that is, they are "highly significant." But in the full multiple regression, the corresponding p-values inflate to .46, .36, and .52, respectively: not significant at all. The reason for this is that once $y$ has been regressed against $x_1$ and $x_2$, the only stuff left to "explain" is the tiny amount of error in the residuals, which will approximate $\varepsilon$, and this error is almost completely unrelated to the remaining $x_i$. ("Almost" is correct: there is a really tiny relationship induced from the fact that the residuals were computed in part from the values of $x_1$ and $x_2$ and the $x_i$, $i=3,4,5$, do have some weak relationship to $x_1$ and $x_2$. This residual relationship is practically undetectable, though, as we saw.)

The conditioning number of the design matrix is only 2.17: that's very low, showing no indication of high multicollinearity whatsoever. (Perfect lack of collinearity would be reflected in a conditioning number of 1, but in practice this is seen only with artificial data and designed experiments. Conditioning numbers in the range 1-6 (or even higher, with more variables) are unremarkable.) This completes the simulation: it has successfully reproduced every aspect of the problem.

The important insights this analysis offers include

  1. p-values don't tell us anything directly about collinearity. They depend strongly on the amount of data.

  2. Relationships among p-values in multiple regressions and p-values in related regressions (involving subsets of the independent variable) are complex and usually unpredictable.

Consequently, as others have argued, p-values should not be your sole guide (or even your principal guide) to model selection.


Edit

It is not necessary for $n$ to be as large as $500$ for these phenomena to appear. Inspired by additional information in the question, the following is a dataset constructed in a similar fashion with $n=24$ (in this case $x_j = 0.4 x_1 + 0.4 x_2 + \delta$ for $j=3,4,5$). This creates correlations of 0.38 to 0.73 between $x_{1-2}$ and $x_{3-5}$. The condition number of the design matrix is 9.05: a little high, but not terrible. (Some rules of thumb say that condition numbers as high as 10 are ok.) The p-values of the individual regressions against $x_3, x_4, x_5$ are 0.002, 0.015, and 0.008: significant to highly significant. Thus, some multicollinearity is involved, but it's not so large that one would work to change it. The basic insight remains the same: significance and multicollinearity are different things; only mild mathematical constraints hold among them; and it is possible for the inclusion or exclusion of even a single variable to have profound effects on all p-values even without severe multicollinearity being an issue.

x1 x2 x3 x4 x5 y
-1.78256    -0.334959   -1.22672    -1.11643    0.233048    -2.12772
0.796957    -0.282075   1.11182 0.773499    0.954179    0.511363
0.956733    0.925203    1.65832 0.25006 -0.273526   1.89336
0.346049    0.0111112   1.57815 0.767076    1.48114 0.365872
-0.73198    -1.56574    -1.06783    -0.914841   -1.68338    -2.30272
0.221718    -0.175337   -0.0922871  1.25869 -1.05304    0.0268453
1.71033 0.0487565   -0.435238   -0.239226   1.08944 1.76248
0.936259    1.00507 1.56755 0.715845    1.50658 1.93177
-0.664651   0.531793    -0.150516   -0.577719   2.57178 -0.121927
-0.0847412  -1.14022    0.577469    0.694189    -1.02427    -1.2199
-1.30773    1.40016 -1.5949 0.506035    0.539175    0.0955259
-0.55336    1.93245 1.34462 1.15979 2.25317 1.38259
1.6934  0.192212    0.965777    0.283766    3.63855 1.86975
-0.715726   0.259011    -0.674307   0.864498    0.504759    -0.478025
-0.800315   -0.655506   0.0899015   -2.19869    -0.941662   -1.46332
-0.169604   -1.08992    -1.80457    -0.350718   0.818985    -1.2727
0.365721    1.10428 0.33128 -0.0163167  0.295945    1.48115
0.215779    2.233   0.33428 1.07424 0.815481    2.4511
1.07042 0.0490205   -0.195314   0.101451    -0.721812   1.11711
-0.478905   -0.438893   -1.54429    0.798461    -0.774219   -0.90456
1.2487  1.03267 0.958559    1.26925 1.31709 2.26846
-0.124634   -0.616711   0.334179    0.404281    0.531215    -0.747697
-1.82317    1.11467 0.407822    -0.937689   -1.90806    -0.723693
-1.34046    1.16957 0.271146    1.71505 0.910682    -0.176185
share|improve this answer
    
Given that I am working on explaining relationships between these variables and their importance in predicting the y, does the lack of collinearity tell me essentially what the initial multiple linear regression told me: that only two variables are important? If the variables did show collinearity, then would it mean that several are important, but provide similar information? Please let me know if I'm completely missing the point - I am by no means a stats expert. –  allie Mar 12 '12 at 19:42
    
Oh, and I'll add this into my original post, but my n is 24 (human subjects work, so that's pretty high). Based on your post, I can assume this is why multicollinearity people suggest getting more data - to better highlight differences. –  allie Mar 12 '12 at 19:42
    
I provide a new example showing how your phenomena can happen even when $n=24$. It could easily be modified so that all numbers involved are positive whole numbers: counts, that is. –  whuber Mar 12 '12 at 21:12
1  
Re your first comment: collinearity suggests some of the explanatory variables (IVs) may be redundant, but this is not necessarily the case. What matters are the relationships among the IVs and the dependent variable (DV). It's possible for one of the IVs to be heavily dependent on the other IVs, yet contain uniquely useful information related to the DV. This is a critical concept: no amount of analysis of relationships among the IVs alone is going to tell you which variables best explain the DV. Lack of collinearity--a property solely of the IVs--doesn't reveal anything about the DV. –  whuber Mar 12 '12 at 21:17

Do I or do I not have a multicollinearity problem? If I do, then how should I proceed?

It's not an either-or situation. And I am skeptical about the "4 or 5" guideline. For each of your predictors, the standard error of the coefficient is between 2.2 and 5.6 times as large as it would be if the predictor were uncorrelated with the others. And the portion of a given predictor that cannot be explained by the others ranges from 1/2.2 to 1/5.6, or 18% to 45%. Altogether, that seems a pretty substantial amount of collinearity.

But let's step back for a minute. Are you really trying to predict *Y*, as opposed to trying to explain it? If the former, then I don't suppose you need care whether the significance level of a given variable changes when others are present in the model. Your job is really much easier than it would be if true explanation were needed.

If explanation is your goal, you'll need to consider the way these variables interrelate--something that requires more than statistical information. Clearly they overlap in the way they relate to Y, and this collinearity will make it difficult to establish, for example, their rank order of importance in accounting for Y. In this situation there's no one clear path for you to follow.

In any case, I hope you are considering methods of crossvalidation.

share|improve this answer
    
This answer, like John's, appears to confuse low p-values with high correlation. Remember: the standard errors of the coefficients decrease with increasing amounts of data (caeteris paribus), so the low p-values can be achieved with data having almost no correlations, provided enough observations are present. –  whuber Mar 12 '12 at 17:39
    
Confuse low p-values with high correlation? Pas du tout! –  rolando2 Mar 12 '12 at 20:55
    
Then please explain how a strictly p-value concept ("the standard error of the coefficient is between 2.2 and 5.6 times as large as it would be if the predictor were uncorrelated with the others") leads you to conclude "that seems a pretty substantial amount of collinearity," which is strongly related to correlation (measures of collinearity are properties of the correlation matrix when the variables are standardized). –  whuber Mar 12 '12 at 21:20
    
I look at it this way. When VIF is 5.6, 82% of the variance in that predictor can be accounted for by the other predictors. I don't see how this could be dependent on N. –  rolando2 Mar 13 '12 at 0:58
1  
As a counterpoint to this pessimistic assessment (which does have some justification in rules of thumb such as requiring 5-10 times as many observations as variables), it is noteworthy that entire fields of modeling and data analysis have grown up around problems that have few observations and many predictors, such as DACE (design and analysis of computer experiments). See ressources-actuarielles.net/EXT/ISFA/1226.nsf/… for a seminal paper. –  whuber Mar 13 '12 at 5:06

You have multicollinearity. Your initial analysis demonstrated that. As far as it being a problem, that's another question that seems to have many answers in your case.

Maybe if you got the basic issue better it would be more obvious what to do?...

With the multicollinearity your regression coefficients are about the unique (well closer to unique) contributions of each variable to your model. If some are correlated with each other then each correlated one's unique contribution is smaller. That's probably partially why none are significant when they're all there together but when used alone they can be.

The first thing you likely need to do is consider what the intercorrelation among your variables means. For example, do you have a bunch of variables that just stand for the same thing? Did you just happen to measure your predictors over a poor scale and get incidental correlations? Don't try to fix the regression, try to understand your variables.

Consider X1 and X2 with a very strong correlation between them, say r = 0.90. If you put X1 in the model and it's a significant predictor then another model with X2 alone will very likely be significant as well because they're almost the same thing. If you put them in the model together at least one of them has to suffer because the multiple regression is going to solve to their unique contributions. They might both be non-significant. But that's not the point, the point is recognizing why they overlap so much and if they even say anything different from one another and whether you need them or not? Maybe one expresses an idea more meaningfully and more related to your response variable than the other. Maybe you'll conclude that they're the same thing with different levels of variability.

Also, when looking at models of any kind, but especially with intercorrelated predictors, p-values are a terrible way to tell if a new predictor makes a meaningful contribution (if that's what you're trying to do... not sure what you're trying to do because it sounds like you're just trying to make the regression either A) simple, or B) come out the way you want... neither of which are feasible). You're probably best off looking at AIC to help you determine which predictors you should keep and which don't contribute anything.

share|improve this answer
    
How do low p-values demonstrate multicollinearity? The two concepts are completely different. With enough data, you can have low p-values and almost no collinearity at all. –  whuber Mar 12 '12 at 17:35
    
This is exactly what I am contesting, John: you have concluded in your first sentence that what the OP describes implies "You have multicollinearity." But this is exactly what the OP wants to know: "do I or do I not have a multicollinearity problem"? I contend that the correct answer is "you haven't given us quite enough information, but probably not" because the phenomena described in the question are perfectly consistent with well-conditioned problems. Indeed, the low VIFs reported by the OP suggest that your assertion is false. –  whuber Mar 12 '12 at 18:19
    
I didn't say that anywhere. Perhaps you mean what I said about the initial analysis. The initial analysis was that the effects change a lot depending on what other effects were added. That was due to multi-collinearity (although it doesn't quantify it). Of course significance is a different issue. I really don't know what you're getting at? –  John Mar 12 '12 at 18:21
    
Sorry Whuber for updating comment, but yours works fine anyway.... readers, the last two above are reversed and it's my fault. Whuber, I was just focused on the word "problem". Multicollinearity is something you quantify. There is some. It suggests thinking hard about the variables regardless. It also suggests that the reason additive predictors are changing when added or removed is due to that multicollinearity. I didn't get the impression the questioner really wanted an answer about it being a calculation "problem". –  John Mar 12 '12 at 18:29
    
It's possible we interpret the question in different ways, John. Because I don't want to leave the issue possibly confused by my comments here, I added a reply to explain my point. –  whuber Mar 12 '12 at 18:58

Personally, I'd use condition indexes and the variance explained table to analyze collinearity.

I would also not use p values as a criterion for model building, and when comparing models with 6 IVs to models with 1, I'd look at changes in the effect size of the parameter for the variable that is both.

But you can certainly have the results you mention without collinearity. Collinearity is only about the X variables and their relationship. But two variables could both relate strongly to Y while not relating strongly to each other.

share|improve this answer
1  
This seems unintuitive to me, that two variables could relate strongly to Y without relating strongly to each other. Is there an example you could point me to, or a longer explanation? –  allie Mar 11 '12 at 13:09
    
@Peter - with 1-(1/5.6) = 82% of the variance in that last predictor explained by the others, why do you say there might not be collinearity? –  rolando2 Mar 11 '12 at 13:29
3  
Allie, that's a good question. Take two unrelated variables $x_1$ and $x_2$, of comparable variances, and form $y = x_1 + x_2$. Now $y$ is strongly related to each of $x_1$ and $x_2$ without $x_1$ and $x_2$ having any relation at all. –  whuber Mar 12 '12 at 19:32

Regarding multicollinearity there are various thresholds being mentioned usually converging around a VIF of 10 corresponding to an underlying R Square value of 0.90 between the tested variable vs the other independent variables. The VIFs of your variables appear passable, and you could technically keep them in a model.

Yet, I would use a stepwise regression method to see which are the best combination of variables and how much more explanation (incremental increase in R Square) you get by adding variables. The arbitrating benchmark should be the Adjusted R Square value that adjusts the R Square value downward by penalizing the model for adding variables.

Your variables are somewhat correlated with each other. This is inevitable, it is just a matter of degree. Given the VIFs you mention, I suspect intuitively that you will get the vast majority of the information/explanation bit from the best 2 variable combination. And, that adding variables may add only marginal incremental value.

When looking at the combination of variables that are selected by the stepwise regression process, I would also look at what variables are selected and if their regression coefficient signs are consistent with their correlation with y. If they are not, it can be due to a legitmate interaction between the variables. But, it could also be a result of model overfitting and that the regression coefficients are spurious. They reflect a mathematical fit, but are meaningless in terms of underlying causality.

Another way to select your variables is to decide from a logic standpoint which ones are the main 2 or 3 variables that should be in the model. You start with those and then check how much more information do you get by adding a variable. Check the adjusted R Square, consistency of the regression coefficient relative to the original regression, and obviously test all the models with hold out period. Pretty soon, it will be evident what is your best model.

share|improve this answer
2  
I disagree that an automatic stepwise selection procedure would be preferable. In such a case you would be selecting based on a random variable, which causes a lot of problems. I discuss this phenomenon here. If stepwise selection were applied anyway, I would recommend using the AIC, instead of $R_{adj}^2$, as the steeper penalty would be more appropriate; however, I do not recommend using stepwise selection. –  gung Mar 11 '12 at 17:30
    
Many of the problems you outline are common to linear regressions model in general. I am not sure that such technical problems are reasons to throw out all stepwise regression methods and linear regression in general. I am unclear why stepwise regression "is selecting based on a random variable, which causes a lot of problems." Stepwise finds the best fit, like any model does. What I think is more important is to ensure that the mathematical fit corresponds to the underlying theory or logic of the problem you are solving for. –  Gaetan Lion Mar 12 '12 at 1:14
2  
I can edit it if need be, but the problems quoted are not about linear models in general, just stepwise selection. I do not advocate throwing out linear regression. Stepwise algorithms return highly biased betas & inaccurate CIs that are largely impossible to correct. $R^2$, $R_{adj}^2$, $F$, $p$, etc. are random variables: if you get more data from the same data generating process & fit an identical model repeatedly, these values will vary. Selecting a model based on them incorporates error in ways that eliminate the value of the model. I do agree w/ using theory & logic to select a model. –  gung Mar 12 '12 at 3:36
    
gung, I am not sure we are talking of the same thing. I am using Stepwise Regression in two ways. One is manual, you build a model using the best regressor. And, you add on to it using the 2nd best variable that best explain the error of the first model. And, you keep going till your AIC score deteriorates. The 2nd method I have used is using XLStat software that automates the process and was developed by Thierry Fahmy and his team. He has I understand a PhD in math along with others on his team. And, I am not confident they would have fallen into all the traps you mentioned. –  Gaetan Lion Mar 12 '12 at 16:28
2  
Gaetan, I think what @gung is trying to say is that stepwise methods may cause severe damage to the initial statistical regression framework (loss function, p-values, etc.). See Frank Harrell's response and comments here. Penalized regression, as discussed in several threads, might be a good alternative. "(...) software that automates the process" reminds me of R fortune(224): no troll here, just want to highlight that you don't necessarily need to trust what software automagically gives (or offers) you as an option. –  chl Mar 12 '12 at 22:49

If your explanatory variables are count data, and it is not unreasonable to assume that they are normally distributed, you can transform them into standard normal variates using the R scale command. Doing this can reduce the collinearity. But that will probably not solve the whole problem.

A useful batch of R commands for analyzing and dealing with collinearity are found on Florian Jaeger's blog, including:

z. <- function (x) scale(x)
r. <- function (formula, ...) rstandard(lm(formula, ...))

The z. function converts a vector into a standard normal variate. The r. function returns standardized residuals for regressing one predictor against another. You can use this to effectively divide the model deviance into different tranches so that only some variables have access to the most senior tranche, then the next tranche will be offered to residualized variables. (Sorry for my homespun terminology) So if a model of the form

Y ~ A + B

suffers from multicollinearity, then you can run either of

Y ~ A + r.(B)
Y ~ r.(A) + B

so that only the residuals of the "junior tranche" variable (when regressed against the "senior tranche" variable) are fitted to the model. This way, you are shielded from multicollinearity, but have a more complicated set of parameters to report.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.