2
$\begingroup$

I want to investigate the correlation of $N$ and $G$ with $X$. In a correlation matrix, $N$ and $G$ are both positively correlated with $X$. $N$ and $G$ are correlated, too, by the way. In a multiple regression however, $G$ was a significant predictor for $X$, while $N$ was not.

Can somebody help me to interpret this?

$\endgroup$
1
  • $\begingroup$ Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. $\endgroup$
    – Community Bot
    Commented Sep 9, 2023 at 9:17

2 Answers 2

7
$\begingroup$

This can happen when N and G are redundant (overlapping, correlated) predictors of X. Once G is in the regression model, N may no longer be needed because N may not account for a unique variance component that is not already accounted for by G. This is one reason why we use regression in the first place--to detect redundancies among predictors.

$\endgroup$
1
  • $\begingroup$ This is not uncommon with observational data, but can be avoided if you run designed experiments which are orthogonal. $\endgroup$
    – pjs
    Commented Sep 10, 2023 at 10:27
9
$\begingroup$

Redundancy

As an example of Christian's point, lets say we create three variables $X$, $Y$, and $Z$ with this following simulated data in R:

set.seed(123)
y <- rnorm(1000)
x <- y + rnorm(1000)
z <- x + y + rnorm(1000)
df <- data.frame(x,y,z)
cor(df)

You can see they are heavily correlated:

          x         y         z
x 1.0000000 0.7314560 0.8833477
y 0.7314560 1.0000000 0.8217593
z 0.8833477 0.8217593 1.0000000

If we then fit that data to a regression:

fit <- lm(y ~ x + z)
summary(fit)

We now see that $X$ is barely influential on $Y$:

Call:
lm(formula = y ~ x + z)

Residuals:
     Min       1Q   Median       3Q      Max 
-1.95208 -0.36139  0.00194  0.36844  1.73213 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept) -0.002207   0.017905  -0.123    0.902    
x            0.017004   0.025878   0.657    0.511    
z            0.317513   0.015289  20.767   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.5655 on 997 degrees of freedom
Multiple R-squared:  0.6754,    Adjusted R-squared:  0.6748 
F-statistic:  1037 on 2 and 997 DF,  p-value: < 2.2e-16

When one variable that is highly correlated with another "cancels out" its influence, we sometimes call this a suppression effect, which you can read about here.

The Opposite

Note that the converse can also sometimes be true, in that two uncorrelated predictors suddenly become statistically significant when entering them into a regression. Two important phenomenons related to this are confounder bias (a predictor either mediates or predicts two variables that otherwise have no causal relationship) and collider bias (a third variable is the combinatory effect of these two unrelated predictors). Simpson's paradox is probably the most well known issue, where a relationship seems to appear to be one direction but is actually the sum effect of many opposite directions of effect. In other words, a set of variables may be highly correlated or completely uncorrelated but because of some of the confounding influences already mentioned, the actual relationship is in reverse. Here is another example in R with the iris dataset, which shows sepal lengths for flowers. Here they appear to have no relationship as is.

iris %>% 
  ggplot(aes(x=Sepal.Width,
             y=Sepal.Length))+
  geom_point()+
  geom_smooth(method = "lm")

enter image description here

But if we check flower species patterns:

iris %>% 
  ggplot(aes(x=Sepal.Width,
             y=Sepal.Length,
             color=Species))+
  geom_point()+
  geom_smooth(method = "lm")

We see sepal width is actually highly associated with sepal length:

enter image description here

Fitting these regressions emulates this issue:

flower1 <- lm(Sepal.Length ~ Sepal.Width, iris)
flower2 <- lm(Sepal.Length ~ Sepal.Width + Species, iris)
summary(flower1)
summary(flower2)

As shown by the summaries below, where sepal width goes from statistically non-significant to significant. Notice the sign change in the point estimate for the coefficient:

> summary(flower1)

Call:
lm(formula = Sepal.Length ~ Sepal.Width, data = iris)

Residuals:
    Min      1Q  Median      3Q     Max 
-1.5561 -0.6333 -0.1120  0.5579  2.2226 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)   6.5262     0.4789   13.63   <2e-16 ***
Sepal.Width  -0.2234     0.1551   -1.44    0.152    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.8251 on 148 degrees of freedom
Multiple R-squared:  0.01382,   Adjusted R-squared:  0.007159 
F-statistic: 2.074 on 1 and 148 DF,  p-value: 0.1519

> summary(flower2)

Call:
lm(formula = Sepal.Length ~ Sepal.Width + Species, data = iris)

Residuals:
     Min       1Q   Median       3Q      Max 
-1.30711 -0.25713 -0.05325  0.19542  1.41253 

Coefficients:
                  Estimate Std. Error t value Pr(>|t|)    
(Intercept)         2.2514     0.3698   6.089 9.57e-09 ***
Sepal.Width         0.8036     0.1063   7.557 4.19e-12 ***
Speciesversicolor   1.4587     0.1121  13.012  < 2e-16 ***
Speciesvirginica    1.9468     0.1000  19.465  < 2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.438 on 146 degrees of freedom
Multiple R-squared:  0.7259,    Adjusted R-squared:  0.7203 
F-statistic: 128.9 on 3 and 146 DF,  p-value: < 2.2e-16
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.