# What should I be aware of when using multiple regression to find "causal" relationships in my data?

First of all, I realize multiple regression does not really give actually "causal" inferences about the data. Let me explain my current case:

I have four independent variables which I hope (but am not sure) are involved in driving the thing I'm measuring. I wanted to use multiple regression to see how much each of these variables are contributing to my dependent variable, and did so. Supposedly, variable "Number four" is influencing my outcome measure very strongly (beta weight close to 0.7).

However, I've been told this isn't enough, because some of my "independent" variables may in fact be correlated with each-other. In that case, I could think "Variable four" is driving my dependent variable, when really both three and four could be contributing equally. This seems correct, but since I'm new to this I'm unsure.

How can I systemically avoid this problem in the future? What specific procedures would you recommend when using multiple regression to make sure that your "independent" data does not already contain hidden correlations?

Edit: The data itself is a series of network (graph) models of a particular neurological state. I'm measuring the "clustering coefficient" which describes the topology of each network as a whole (dependent variable here), and then seeing if the individual connectivities of four nodes in the larger 100+ network are driving the global clustering values (four independent variables). However, these nodes are part of a network, so sort of by definition it's possible they're correlated to some extent.

• What makes a correlation causal is a contentious issue in philosophy of science. The "gold standard" is to perform an experiment where the treatment variable is assigned at random, thus ensuring other potentially confounding covariates are not systematically related to the treatment. However, in many fields and for many questions, experiments are impossible. Some turn to so-called natural experiments for exogeneity. You may be interested in Paul Holland's classic article, "Statistics and causal inference". Journal of the American Statistical Association, 81, 945-970. Apr 16, 2011 at 21:02
• You're asking very important questions, but it's doubtful anyone could give you a definitive series of steps to take or a nice, condensed recipe; mastering this issue is a long-term proposition. Additional suggestions about terms and topics to study: suppressor variables; tolerance and variance inflation estimates; zero-order, partial, and semipartial (part) correlations; variable selection methods; crossvalidation. Apr 17, 2011 at 0:28
• If you were to tell us the purpose of this modeling, you might get even more helpful suggestions. Multiple regression deals quite well with correlated independent variables, as long as they aren't too highly correlated, resulting in multicolinearity. As others have said, assessing causality is difficult (but not impossible) outside of a randomized experiment. See some of these links: delicious.com/MichaelBishop/causality for more on that topic. Apr 20, 2011 at 0:10
• The data itself is a series of network (graph) models of a particular neurological state. I'm measuring the "clustering coefficient" which describes the topology of each network as a whole (dependent variable here), and then seeing if the individual connectivities of four nodes in the larger 100+ network are driving the global clustering values (four independent variables). However, these nodes are part of a network, so sort of by definition it's possible they're correlated to some extent. Apr 20, 2011 at 4:27
• you can only test hypothesis about causality statistically on your timeseries with Granger Causality test to detect if one timeseries follows the other (probably being caused by that last one) Feb 29 at 17:14

You cannot "systemically avoid this problem in the future", because it should not be called a "problem". If the reality of the material world features strong covariates, then we should accept it as fact and adjust our theories and models in consequence. I like the question very much, and hope that what follows will not sound too disappointing.

Here are some adjustments that might work for you. You will need to review a regression handbook before proceeding.

• Diagnose the issue, using correlation or post-estimation techniques like the Variance Inflation Factor (VIF). Use the tools mentioned by Peter Flom if you are using SAS or R. In Stata, use pwcorr to build a correlation matrix, gr matrix to build a scatterplot matrix, and vif to detect problematic tolerance levels of 1/VIF < 0.1.

• Measure the interaction effect by adding, for example, var3*var4 to the model. The coefficient will help you realise how much is at play between var3 and var4. This will only bring you so far as partially measuring the interaction, but it will not rescue your model from its limitations.

• Most importantly, if you detect strong multicollinearity or other issues like heteroscedasticity, you should ditch your model and start again. Model misspecification is the plague of regression analysis (and frequentist methods in general). Paul Schrodt has several excellent papers on the issue, including his recent "Seven Deadly Sins" that I like a lot.

This answers your point on multicollinearity, and a lot of this can be learnt from the regression handbook over at UCLA Stat Computing. It does not answer your question on causality. Briefly put, regression is never causal. Neither is any statistical model: causal and statistical information are separate species. Read selectively from Judea Pearl (example) to learn more on the matter.

All in all, this answer does not cancel out the value of regression analysis, or even of frequentist statistics (I happen to teach both). It does, however, reduce their scope of appropriateness, and also underlines the crucial role of your initial explanatory theory, which really determines the possibility of your model possessing causal properties.

• +1 for the links to the handbook, mentioning collinearity and IVF, and the specific solutions and even implementations in R. I am curious to hear your opinion about whether the data itself is not suited to regression analysis- I edited the question above to reflect that these are measurements of a network. Apr 20, 2011 at 4:39
• Sorry for the late reply, but I know unfortunately too little on the topic anyway to answer that you are using the right technique. My guess is that SNA contains other tools that will help (e.g. model different centrality measures when you suppress any combination of your four nodes).
– Fr.
May 8, 2011 at 12:08

If you want to see if the independent variables are correlated, that's easy - just test the correlations e.g. with PROC CORR in SAS, or cor in R, or whatever in whatever package you use.

But that's only part of the problem for causation. More problematic is that some variable that is NOT in your data is involved. Classic examples:

Students who hire tutors get worse grades than students who do not hire tutors.

The amount of damage done by a fire is highly related to the number of firemen who show up.

and (my favorite)

if you regress IQ on astrological sign and age among children age 5 - 12, there is a significant interaction and a significant effect of sign on IQ, but only in young children.

Reasons: 1. Yes. Because students who get really good grades tend not to hire tutors in the first place

1. Yes, because bigger fires do more damage and bring more firemen

2. The amount of school (in months) a kid has had depends on birth month. School systems have age cutoffs. So, one 6 year old may have had 11 months more school than another 6 year old.

And all that is without getting into philosophy!

The relationship between causation and association is basically in answering the following question:

What else, besides the hypothesised causal relationship, could have caused $X$ and $Y$ to be related to each other?

As long as the answer to this question is not "nothing" then you can only talk definitively about association. There can always be that one proposed "causal" relationship is actually a special case of the "correct" causal relationship - this is what happened between Newton's and Einstein's theory of gravity I think. Newton's causal relationship was a special case of Einstein's theory. And his theory will probably be a special case of some other theory.

Additionally, any error at all in your data removes any chance of a definite causal relationship. This is because the phrase "A causes B" is somewhat of a deductive link between A and B. All you have to do to disprove this hypothesis is to find 1 case where B is not present but A is present (for then A is true, but this should mean that B is also true - but we observed B false).

In a regression setting, it is much more constructive to think of prediction than of interpreting coefficients when looking at causation. So if you really do have a good reason to think that variable four is the "main cause" of variable $Y$ (your dependent variable), then you should be able to predict $Y$ with near certainty using variable four. If you cannot do this, then it is inappropriate to concluded that variable four causes $Y$. But if you can do this prediction to near certainty using all four variables - then this is indicating that particular combinations are "causing" $Y$. And whenever you propose a causal relationship you will almost certainly have to "prove it" by reproducing your results with new data - you will need to be able to predict what data will be seen, and be correct about it.

You also need some kind of physical theory about the "causal mechanism" (when I push that button, the light comes on, when I push this button, the light changes color, etc.). If all you have is that the "regression coefficient was 0.7" this does little for establishing a causal mechanism which is at work.

I'm not sure what field your work is in, so this may or may not be of any help- but I'm most familiar with using SPSS with psychological constructs. In my experience, if I have a few variables predicting an outcome variable (or dependent variable) in a regression, and I have one or more independent variables show up as significant predictors, the next step is to see which ones are more incrementally important than others. One way to approach this is with hierarchical regression. This basically answers the question "If I already have 'variable four' to predict my outcome variable, do any of the other variables provide a statistically significant increase in predictive power?" SPSS has a pretty clear way of analyzing this, as I'm sure R and SAS do as well. So, I think hierarchical regression might be your next step in finding out if 'variable four' really is your best bet in predicting your outcome factor. The others who've responded have provided a good discussion of the issues in correlation-causation, so I'll leave that alone... Good luck!