# Influence of correlation in linear regression

I have an output $Y$ and some input values $X_1, \dots X_p$, where the number of variables are smaller than the number of observations ($p<<n$).

I want to understand which of the variables have an influence on the $Y$.

A classical way to do that is to fit a linear model and then check if the $p_{\text{value}}$ is small enough.

There is a variable $X_1$ that is known to have an effect on the output. There is another variable $X_2$ correlated with $X_1$ and I want to know if $X_2$ has actually an influence on the output or is correlated with it only for its correlation with $X_1$.

For example, consider this code in R:

n = 100
x1 = rnorm(n)
x2_0 = rnorm(n, 0, 1)
x2 = x2_0 + 0.3 * x1
cor(x1, x2)

y = x1 + rnorm(n,0,2)

l = glm(y~x1 + x2)
summary(l)

Call:
glm(formula = y ~ x1 + x2)

Deviance Residuals:
Min       1Q   Median       3Q      Max
-7.0096  -1.1390  -0.0377   1.0572   4.3845

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)  0.08310    0.19572   0.425 0.672089
x1           0.73738    0.21698   3.398 0.000985 ***
x2           0.03093    0.19543   0.158 0.874578
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for gaussian family taken to be 3.719181)

Null deviance: 408.13  on 99  degrees of freedom
Residual deviance: 360.76  on 97  degrees of freedom
AIC: 420.09

Number of Fisher Scoring iterations: 2


From the above code we can see that $X_2$ is not important for the $Y$ while $X_1$ is.

I am wondering if this is enough to prove that $X_2$ has or has not an influence on the $Y$ or if there are cases where this method can lead to errors.

I am particularly interested in a case where $X_2$ has a significant $p_{\text{value}}$, but it is not actually related to the output.

• Correlation does not imply causation. You typically need some "economic" or theoretical arguments besides the regression coefficients to argue that a variable does not matter even though its coefficient in the regression appears statistically significant. Here is a whole website full of examples dedicated for correlations that have nothing to do with causation. In short, regression only is not enough to answer the causation question. – Richard Hardy Dec 1 '14 at 19:15
• I agree that correlation does not imply causation. Trough domain knowledge we think that the variable $x_2$ is relevant but we would like to prove it from a statistical point of view. My idea was to show that the presence of the variable $x_2$ reduces the prediction error of the model. The problem is that our model is incredibly weak so I am not sure that this is the best way to proceed. Do you think that regression + domain knowledge would be enough? (I do not think so but some people do) – Donbeo Dec 1 '14 at 19:48
• I would not like to generalize too much, but roughly speaking, domain knowledge + regression seems enough to me. You could compare the BIC of the model with $x_2$ to BIC of the model without $x_2$. If the BIC prefers to include $x_2$, that would be a good argument to keep it. Using only statistical significance test (as an alternative to BIC) would neglect the size of the coefficient. Given enough data, even irrelevant variables tend to become statistically significant, even though coefficient values may be pretty small. Using BIC instead of significance testing should solve this problem. – Richard Hardy Dec 1 '14 at 20:12

The following is a causal question (emphases added):

I want to understand which of the variables have an influence on the Y.

A classical way to do that is to fit a linear model and then check if the pvalue is small enough.

There is a variable X1 that is known to have an effect on the output. There is another variable X2 correlated with X1 and I want to know if X2 has actually an influence on the output or is correlated with it only for its correlation with X1.

Clearly, in your case X2 is not causal because the relevant causal diagram (leaving out independent noise terms) is:

Y <- X1 -> X2


Using this notation, if the system were instead

X2 -> Y <- X1


then X2 would be causal.

Unfortunately, without knowing what direction the arrows point these two causal stories are observationally indistinguishable -- both structures can be set up to the the same matrix of correlations between the three variables and so the same linear regression coefficients.

They differ, of course, in their implied counterfactuals: if you were to step in and manipulate X2 in the first system, Y would not be affected, whereas in the second system it would be.

Consequently, 'domain knowledge' amounts to knowing that your dealing with a diagram like the second one rather than like the first one. (Experimentation would also give you this information).

In the absence of such extra information, the data cannot in general tell you which one you're dealing with.

From the above code we can see that X2 is not important for the Y while X1 is.

You are seeing that X2 is not as useful for predicting Y as X1 is. But that's not your question at all.

I am wondering if this is enough to prove that X2 has or has not an influence on the Y or if there are cases where this method can lead to errors.

In the absence of some assumption about the causal diagram, it's never enough. Given such information, it might be enough. If it is enough, then linear regression might be rather a good way of estimating the size and direction of the true effect.

There are, of course, a lot more causal structures than these. For example, here is, in some sense, an opposite error. If the real diagram is

Y <- X1 <- X2


then X1 and X2 are still correlated and X2 really is a cause of Y by virtue of causing X1, but if you put X1 in your regression then it will look like X2 is not (because X2 is independent of Y conditional on X1).

• I am particularly worried about the case Y <- X1 -> X2 and if from a statistical point of view it can seems that X2->Y even if it is not actually true but it follows from its correlation with $Y$. I am quite sure that with classical statistical techniques based on correlation a situation like this can arise. I was so wondering if the problem can be avoided using a more modern technique like the LASSO or the conditional importance of random forest biomedcentral.com/1471-2105/9/307 – Donbeo Dec 3 '14 at 8:51
• It can't 'be seen' from the statistical point of view, ie by analysing the associations/correlations between variables. Lasso and suchlike are answering a totally different question. They ask: 'if the system were X2 -> Y <- X1 and there was a coefficient for the X2 -> Y part so that that coefficient would be the effect of X2 on Y, then what would its value be?' But you're worried about whether the premises of this question are false. If they are false, the answer is not useful to you. – conjugateprior Dec 3 '14 at 9:17
• Your worrisome situation does happen often, eg when X2 is a proxy for or measurement of X1. (When X1 is unobserved and X2 is a bank of survey questions whose intercorrelations are due to having X1 as a common cause, then you have a measurement model setup) – conjugateprior Dec 3 '14 at 9:20
• I am not worried about this last case. We have both the measurement of X1 and X2`. – Donbeo Dec 3 '14 at 9:35

Correlation does not imply causation. You typically need some "economic" or theoretical arguments besides the regression coefficients to argue that a variable does not matter even though its coefficient in the regression appears statistically significant. Here is a whole website full of examples dedicated for correlations that have nothing to do with causation.

In short, regression only is not enough to answer the causation question.

Trough domain knowledge we think that the variable $x_2$ is relevant but we would like to prove it from a statistical point of view. (a comment)

I would not like to generalize too much, but an argument based on domain knowledge + regression results seems fair enough to me. In practice, you could compare the BIC value of the model with $x_2$ to the BIC value of the model without $x_2$. If BIC suggests including $x_2$, that would be a good argument to keep it. Using only statistical significance test (as an alternative to BIC) would neglect the size of the coefficient. Given enough data, even irrelevant variables tend to become statistically significant, even though coefficient values may be pretty small. Using BIC instead of significance testing should solve this problem.