40
$\begingroup$

X and Y are not correlated (-.01); however, when I place X in a multiple regression predicting Y, alongside three (A, B, C) other (related) variables, X and two other variables (A, B) are significant predictors of Y. Note that the two other (A, B) variables are significantly correlated with Y outside of the regression.

How should I interpret these findings? X predicts unique variance in Y, but since these are not correlated (Pearson), it is somehow difficult to interpret.

I know of opposite cases (i.e., two variables are correlated but regression is not significant) and those are relatively simpler to understand from a theoretical and statistical perspective. Note that some of the predictors are quite correlated (e.g., .70) but not to the extent that I would expect substantial multicollinearity. Maybe I am mistaken, though.

NOTE: I asked this question previously and it was closed. The rational was that this question is redundant with the question "How can a regression be significant yet all predictors be non-significant?". Perhaps I do not understand the other question, but I believe these are entirely separate questions, both mathematically and theoretically. My question is entirely independent from if "a regression is significant". Furthermore, several predictors are significant, while the other question entails variables not being significant, so I don't see the overlap. If these questions are redundant for reasons I do not understand, please insert a comment prior to closing this question. Also, I was hoping to message the moderator who closed the other question to avoid identical questions, but I couldn't find an option to do so.

$\endgroup$
5
  • 2
    $\begingroup$ I do think this is very similar to the previous question. If X and Y are essentially uncorrelated then in a simple linear regression the slope coefficient for X will not be significant. After all the slope estimate is proportional to the sample correlation. Nut multiple regression can be a different story because X and Z together may explain a lot of the variability in Y. Since my answer sounds similar to the answers to the previous question maybe that indicates a distinct similarity. $\endgroup$ Aug 7, 2012 at 21:16
  • 2
    $\begingroup$ Thank you for your reply and very detailed answer in the other thread. I will need to read it over a few time to get the thesis of it. My other concern, I suppose, is how to interpret it practically rather than perhaps statistically or mathematically. Let's say for example swimming speed and trait anxiety are not correlated, but trait anxiety is a significant predictor of swimming speed in a multiple regression alongside other predictors. How can this make sense, practically? Lets say you were writing this in the discussion section of a clinical journal! $\endgroup$
    – Behacad
    Aug 7, 2012 at 21:49
  • 3
    $\begingroup$ @jth Since you maintain the two questions are sufficiently different not to be considered duplicates, please feel free to move your answer to the other one over to here. (I apologize for not originally appreciating the difference.) The new note, I believe, is incorrect in supposing the questions are mathematically different--@Michael Chernick points out they are basically the same--but the emphasis on interpretation establishes a valid reason to keep the threads separate. $\endgroup$
    – whuber
    Aug 7, 2012 at 22:28
  • 1
    $\begingroup$ I also moved the answer here. I think that both questions are quite different but might share some common explanations. $\endgroup$
    – JDav
    Aug 8, 2012 at 0:07
  • 1
    $\begingroup$ This webpage has another great discussion of related topics. It's long, but very good & can help you understand the issues. I recommend reading it completely. $\endgroup$ Aug 9, 2012 at 17:30

4 Answers 4

45
$\begingroup$

Causal theory offers another explanation for how two variables could be unconditionally independent yet conditionally dependent. I am not an expert on causal theory and am grateful for any criticism that will correct any misguidance below.

To illustrate, I will use directed acyclic graphs (DAG). In these graphs, edges ($-$) between variables represent direct causal relationships. Arrowheads ($\leftarrow$ or $\rightarrow$) indicate the direction of causal relationships. Thus $A \rightarrow B$ infers that $A$ directly causes $B$, and $A \leftarrow B$ infers that $A$ is directly caused by $B$. $A \rightarrow B \rightarrow C$ is a causal path that infers that $A$ indirectly causes $C$ through $B$. For simplicity, assume all causal relationships are linear.

First, consider a simple example of confounder bias:

confounder

Here, a simple bivariable regression will suggest a dependence between $X$ and $Y$. However, there is no direct causal relationship between $X$ and $Y$. Instead, both are directly caused by $Z$, and in the simple bivariable regression, observing $Z$ induces a dependency between $X$ and $Y$, resulting in bias by confounding. However, a multivariable regression conditioning on $Z$ will remove the bias and suggest no dependence between $X$ and $Y$.

Second, consider an example of collider bias (also known as Berkson's bias or berksonian bias, of which selection bias is a special type):

collider

Here, a simple bivariable regression will suggest no dependence between $X$ and $Y$. This agrees with the DAG, which infers no direct causal relationship between $X$ and $Y$. However, a multivariable regression conditioning on $Z$ will induce a dependence between $X$ and $Y$, suggesting that a direct causal relationship between the two variables may exist when in fact, none exist. The inclusion of $Z$ in the multivariable regression results in collider bias.

Third, consider an example of incidental cancellation:

cancellation

Let us assume that $\alpha$, $\beta$, and $\gamma$ are path coefficients and that $\beta = -\alpha\gamma$. A simple bivariable regression will suggest no dependence between $X$ and $Y$. Although $X$ is, in fact, a direct cause of $Y$, the confounding effect of $Z$ on $X$ and $Y$ incidentally cancels out the effect of $X$ on $Y$. A multivariable regression conditioning on $Z$ will remove the confounding effect of $Z$ on $X$ and $Y$, allowing for the estimation of the direct effect of $X$ on $Y$, assuming the DAG of the causal model is correct.

To summarize:

Confounder example: $X$ and $Y$ are dependent in bivariable regression and independent in multivariable regression conditioning on confounder $Z$.

Collider example: $X$ and $Y$ are independent in bivariable regression and dependent in multivariable regression conditioning on collider $Z$.

Incidental cancellation example: $X$ and $Y$ are independent in bivariable regression and dependent in multivariable regression conditioning on confounder $Z$.

Discussion:

The results of your analysis are not compatible with the confounder example but are compatible with both the collider example and the incidental cancellation example. Thus, a potential explanation is that you have incorrectly conditioned on a collider variable in your multivariable regression and have induced an association between $X$ and $Y$ even though $X$ is not a cause of $Y$ and $Y$ is not a cause of $X$. Alternatively, you might have correctly conditioned on a confounder in your multivariable regression that was incidentally cancelling out the true effect of $X$ on $Y$ in your bivariable regression.

I find using background knowledge to construct causal models helpful when considering which variables to include in statistical models. For example, if previous high-quality randomized studies concluded that $X$ causes $Z$ and $Y$ causes $Z$, I could make a strong assumption that $Z$ is a collider of $X$ and $Y$ and not condition upon it in a statistical model. However, if I merely had an intuition that $X$ causes $Z$, and $Y$ causes $Z$, but no strong scientific evidence to support my intuition, I could only make a weak assumption that $Z$ is a collider of $X$ and $Y$, as human intuition has a history of being misguided. Subsequently, I would be skeptical of inferring causal relationships between $X$ and $Y$ without further investigating their causal relationships with $Z$. In lieu of or in addition to background knowledge, there are also algorithms designed to infer causal models from the data using a series of tests of association (e.g. PC algorithm and FCI algorithm, see TETRAD for Java implementation, PCalg for R implementation). These algorithms are very interesting, but I would not recommend relying on them without a strong understanding of the power and limitations of causal calculus and causal models in causal theory.

Conclusion:

Contemplation of causal models does not excuse the investigator from addressing the statistical considerations discussed in other answers here. However, I feel that causal models can nevertheless provide a helpful framework when thinking of potential explanations for observed statistical dependence and independence in statistical models, especially when visualizing potential confounders and colliders.

Further reading:

Gelman, Andrew. 2011. "Causality and Statistical Learning." Am. J. Sociology 117 (3) (November): 955–966.

Greenland, S, J Pearl, and J M Robins. 1999. “Causal Diagrams for Epidemiologic Research.” Epidemiology (Cambridge, Mass.) 10 (1) (January): 37–48.

Greenland, Sander. 2003. “Quantifying Biases in Causal Models: Classical Confounding Vs Collider-Stratification Bias.” Epidemiology 14 (3) (May 1): 300–306.

Pearl, Judea. 1998. Why There Is No Statistical Test For Confounding, Why Many Think There Is, And Why They Are Almost Right.

Pearl, Judea. 2009. Causality: Models, Reasoning and Inference. 2nd ed. Cambridge University Press.

Spirtes, Peter, Clark Glymour, and Richard Scheines. 2001. Causation, Prediction, and Search, Second Edition. A Bradford Book.

Update: Judea Pearl discusses the theory of causal inference and the need to incorporate causal inference into introductory statistics courses in the November 2012 edition of Amstat News. His Turing Award Lecture, entitled "The mechanization of causal inference: A 'mini' Turing Test and beyond" is also of interest.

$\endgroup$
8
  • $\begingroup$ The causal arguments are certainly valid but for researcher to subscribe to that approach requires very good knowledge of the underlying phenomena. I wonder if the analysis @Behacad is performing is only exploratory. $\endgroup$
    – JDav
    Aug 8, 2012 at 0:12
  • 1
    $\begingroup$ @Behacad : As mentioned in my answer, I suggest you to forget about the single $\rho$ as your problem is a multivariate one and not bivariate. To measure the influence of your variable of interest, you need to control for other sources of variation that may distort x's measured influence. $\endgroup$
    – JDav
    Aug 8, 2012 at 0:59
  • 5
    $\begingroup$ +1 The illustrations and explanations are very clear and well done. Thank you for the effort and research that (obviously) went into this answer. $\endgroup$
    – whuber
    Aug 8, 2012 at 13:09
  • 1
    $\begingroup$ Also, could someone give me a practical example of "Third, consider an example of incidental cancellation?". The question of causation comes up. If X and Y are not correlated (i.e., changes in X are not associated with changes in Y"), how could we consider this "cause". This is exactly what I am wondering in another question! stats.stackexchange.com/questions/33638/… $\endgroup$
    – Behacad
    Aug 9, 2012 at 0:29
  • 4
    $\begingroup$ It's worth noting that there are some alternative names for these: Confounder -> Common Cause Model; Collider -> Common Effect Model; & Incidental Cancellation is a special case of Partial Mediation. $\endgroup$ Aug 9, 2012 at 17:26
26
$\begingroup$

I think @jthetzel's approach is the right one (+1). In order to interpret these results you will have to think about / have some theory of why the relationships manifest as they do. That is, you will need to think about the pattern of causal relationships that underlies your data. You need to recognize that, as @jthetzel points out, your results are consistent with several different data generating processes. I don't think that any amount of additional statistical tests on the same dataset will allow you to distinguish amongst those possibilities (although further experiments certainly could). So thinking hard about what's known about the topic is vital here.

I want to point out another possible underlying situation that could generate results like yours: Suppression. This is more difficult to illustrate using the arrow diagrams, but if I can augment them slightly, we could think of it like this:

enter image description here

What's important about this situation is that the $\text{Other Variable}$ is made up of two parts, an unrelated ($\text{U}$) part, and a related ($\text{R}$) part. The $\text{Suppressor}$ will be uncorrelated with $\text{Y}$, but may very well be 'significant' in a multiple regression model. Furthermore, the $\text{Other Variable}$ may or may not be 'significantly' correlated with the $\text{Suppressor}$ or $\text{Y}$ on its own. Moreover, your variable X could be playing the role of either the $\text{Suppressor}$ or the $\text{Other Variable}$ in this situation (and thus, again, you need to think about what the underlying pattern might be based on your knowledge of the area).

I don't know if you can read R code, but here's an example I worked up. (This particular example fits better with X playing the role of the $\text{Suppressor}$, but both are not 'significantly' correlated with $\text{Y}$; it should be possible to get the correlation between the $\text{Other Variable}$ and $\text{Y}$ close to 0 and match the other descriptives with just the right settings.)

set.seed(888)                            # for reproducibility

S  =         rnorm(60, mean=0, sd=1.0)   # the Suppressor is normally distributed
U  = 1.1*S + rnorm(60, mean=0, sd=0.1)   # U (unrelated) is Suppressor plus error
R  =         rnorm(60, mean=0, sd=1.0)   # related part; normally distributed
OV = U + R                               # the Other Variable is U plus R
Y  = R +     rnorm(60, mean=0, sd=2)     # Y is R plus error

cor.test(S, Y)                           # Suppressor uncorrelated w/ Y
# t = 0.0283, df = 58, p-value = 0.9775
# cor 0.003721616 

cor.test(S, OV)                          # Suppressor correlated w/ Other Variable
# t = 8.655, df = 58, p-value = 4.939e-12
# cor 0.7507423

cor.test(OV,Y)                           # Other Var not significantly cor w/ Y
# t = 1.954, df = 58, p-value = 0.05553
# cor 0.2485251

summary(lm(Y~OV+S))                      # both Suppressor & Other Var sig in mult reg
# Coefficients:
#              Estimate Std. Error t value Pr(>|t|)   
# (Intercept)   0.2752     0.2396   1.148  0.25557   
# OV            0.7232     0.2390   3.026  0.00372 **
# S            -0.7690     0.3415  -2.251  0.02823 * 

My point here isn't that this situation is the one that underlies your data. I don't know if this is more or less likely than the options @jthetzel suggests. I only offer this as more food for thought. To interpret your current results, you need to think about these possibilities and decide what makes the most sense. To confirm your choice, careful experimentation will be needed.

$\endgroup$
2
  • 3
    $\begingroup$ Excellent! Thank you. This serves as another good example of what could be happening in my data. Seems like I can only accept one answer, though... $\endgroup$
    – Behacad
    Aug 10, 2012 at 16:06
  • $\begingroup$ No problem, @Behacad, I think jthetzel deserves the check mark; I'm just happy to help. $\endgroup$ Aug 10, 2012 at 16:10
9
$\begingroup$

Just some visualization that it is possible.

On picture (a) "normal" or "intuitive" regressional situation is shown. This pic is the same as for example found (and explained) here or here.

The variables are drawn as vectors. Angles between them (their cosines) are the variables' correlations. $Y'$ here designates the variable of predicted values (more often notated as $\hat Y$). Skew coordinate of its edge onto a predictor vector (skew projection, parallel to the other predictor) - notch $b$ - is proportional to the regression coefficient of that predictor.

On pic (a), all three variables correlate positively, and both $b_1$ and $b_2$ are also positive regression coefficients. $X_1$ and $X_2$ "compete" in the regression, with the regression coefficients being their score in that contest.

enter image description here

On picture (b) shown is situation where predictor $X_1$ correlates with $Y$ positively, still it's regression coefficient is zero: the endpoint of the prediction $Y'$ projects at the origin of vector $X_1$. Note that this fact coincides with that $Y'$ and $X_2$ superimpose, which means that the predicted values absolutely correlate with that other predictor.

On picture (c) is the situation where $X_1$ does not correlate with $Y$ (their vectors are orthogonal), yet the regression coefficient of the predictor is not zero: it is negative (the projection falls behind $X_1$ vector).

Data and analysis approximately corresponding to pic (b):

       y       x1       x2
1.644540 1.063845  .351188
1.785204 1.203146  .200000
-1.36357 -.466514 -.961069
 .314549 1.175054  .800000
 .317955  .100612  .858597
 .970097 2.438904 1.000000
 .664388 1.204048  .292670
-.870252 -.993857 -1.89018
1.962192  .587540 -.275352
1.036381 -.110834 -.246448
 .007415 -.069234 1.447422
1.634353  .965370  .467095
 .219813  .553268  .348095
-.285774  .358621  .166708
1.498758 -2.87971 -1.13757
1.671538 -.310708  .396034
1.462036  .057677 1.401522
-.563266  .904716 -.744522
 .297874  .561898 -.929709
-1.54898 -.898084 -.838295

enter image description here

Data and analysis approximately corresponding to pic (c):

       y       x1       x2
1.644540 1.063845  .351188
1.785204 -1.20315  .200000
-1.36357 -.466514 -.961069
 .314549 1.175054  .800000
 .317955 -.100612  .858597
 .970097 1.438904 1.000000
 .664388 1.204048  .292670
-.870252 -.993857 -1.89018
1.962192 -.587540 -.275352
1.036381 -.110834 -.246448
 .007415 -.069234 1.447422
1.634353  .965370  .467095
 .219813  .553268  .348095
-.285774  .358621  .166708
1.498758 -2.87971 -1.13757
1.671538 -.810708  .396034
1.462036 -.057677 1.401522
-.563266  .904716 -.744522
 .297874  .561898 -.929709
-1.54898 -1.26108 -.838295

enter image description here

Observe that $X_1$ in the last example served as suppressor. Its zero-order correlation with $Y$ is practically zero but its part correlation is much larger by magnitude, $-.224$. It strengthened to some extent the predictive force of $X_2$ (from $.419$, a would-be beta in simple regression with it, to beta $.538$ in the multiple regression).

$\endgroup$
1
  • $\begingroup$ Thanks! It still feels somewhat counterintuitive, but at least your pictures show it's feasible :) $\endgroup$ Aug 11, 2017 at 8:28
7
$\begingroup$

I agree with the previous answer but hope I can contribute by giving more details.

The correlation coefficient is just measuring the linear dependence between $X$ and $Y$ and it's not controlling for the fact that other variables might be involved in the relationship as well. In fact the correlation coefficient equals the slope parameter of the following regression scaled by $x$ and $y$ standard deviations :

$Y = a + \beta x + u$

where $\hat \rho_{yx} = \hat \beta \hat\sigma_x/\hat\sigma_y$

But what happens if $Y$ is generated by other variables as well, thus the real model is something like:

$Y = a + \beta x + \sum_j\alpha_jz_j + u$

Under this real model, it becomes obvious that estimating the first one (only with x) will yield a biased $\beta$ estimate as that model is omitting the $z_j$ regressors(this implies that $\rho$ is also biased !). So your results are in line with the fact that the omitted variables are relevant. To deal with this issue , theory on correlation analysis provides the partial correlation coefficient (I'm sure you will find references on this) which basically calculates $\rho_{xy|z}$ from the latter estimating equation that controls for $z_j$.

$\endgroup$
1
  • $\begingroup$ $\rho$ biased means that its value is unreliable, it could be anything from -1 to 1. If you accept to give an interpretation to it , then you are implicitly assuming your universe has 2 variables of interest only. If you suspect there might be others, why to calculate a bivariate $\rho$ ? e.g. a universe must be defined before starting the analysis and yours is multivariate (>2) From that point of view, a bivariate analysis suffers from an omitted variables issue. $\endgroup$
    – JDav
    Aug 8, 2012 at 0:31

Not the answer you're looking for? Browse other questions tagged or ask your own question.