2
$\begingroup$

Suppose we have have 3 variables: Y , X , Z.

then we have :

Simple Regression : $ Y = aX+b$

Multiple Regression : $Y = aX + bZ + c$

I know that in simple regression we can have a low p-value(showing that there is a relationship between X and Y) and high p-value in Multiple Linear Regression which is caused by collinearity between X , Z .

But is it possible the other way? I mean is it possible to have high p-value in Simple regression and low p-value in Multiple Regression? If it is possible, what would the reason be?

Thanks.

$\endgroup$

1 Answer 1

1
$\begingroup$

YES

In some sense, this is why we do regression, to reduce the variance of the conditional distribution and, hopefully, allow the signal of the effect of interest to cut through the reduced noise.

Consider the following situation where we want to know if the color group g affects y.

set.seed(2022)
N <- 50
x <- seq(0, 100, 100/(N - 1))
g <- rep(c(0, 1), N/2)
y <- x + g + rnorm(N)

enter image description here

The two groups look close together, and a regression on just the color variable shows color not to be significant.

L1 <- lm(y ~ g)
summary(L1)

Call:
lm(formula = y ~ g)

Residuals:
    Min      1Q  Median      3Q     Max 
-49.990 -25.091  -0.475  25.447  49.511 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)   48.885      6.025   8.114 1.48e-10 ***
g              2.972      8.520   0.349    0.729    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 30.12 on 48 degrees of freedom
Multiple R-squared:  0.002528,  Adjusted R-squared:  -0.01825 
F-statistic: 0.1217 on 1 and 48 DF,  p-value: 0.7288

However, we know how y is constructed and can see that g absolutely has a linear effect on y. If we consider the covariate x, we reduce the conditional variance from about $30$ to about $1$, allowing the effect of g to be apparent.

L2 <- lm(y ~ g + x)
summary(L2)

Call:
lm(formula = y ~ g + x)

Residuals:
     Min       1Q   Median       3Q      Max 
-2.64091 -0.69691  0.07407  0.55674  1.38453 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept) -0.209967   0.274293  -0.765 0.447809    
g            0.926162   0.251351   3.685 0.000591 ***
x            1.002360   0.004267 234.893  < 2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.8881 on 47 degrees of freedom
Multiple R-squared:  0.9992,    Adjusted R-squared:  0.9991 
F-statistic: 2.766e+04 on 2 and 47 DF,  p-value: < 2.2e-16
$\endgroup$
4
  • $\begingroup$ Thank you for your answer. Your example is OK, but I have not understood the cause of such a behavior. $\endgroup$
    – Mohammad
    Jul 1, 2022 at 14:51
  • $\begingroup$ @Mohammad What do you mean by the "cause" of such a behavior? $\endgroup$
    – Dave
    Jul 1, 2022 at 14:52
  • $\begingroup$ As I mentioned in my question, collinearity between X , Z causes p-value of X (or Z) in multiple regression to be high and in simple regression to be low.(for example the relationship among temperatures, Ice cream sales and Shark attacks at beach which is described in ISLR book). now I want to know what's happening when we have high p-value in simple regression and low p-value in multiple regression ? $\endgroup$
    – Mohammad
    Jul 1, 2022 at 15:00
  • $\begingroup$ We have a certain amount of effect (signal). In order to see the effect, it has to stand out from the background. To do this, we either need to make a more intense signal or reduce the background. By including an additional variable that has an impact on the outcome, we can account for that, rather than considering it background, and allow an equal signal to stand out against a dimmer background. // Consider talking somewhere loud. To hear each other, you either speak loudly or move somewhere quieter. Regression is analogous to moving somewhere quieter, allowing your normal voice to stand out. $\endgroup$
    – Dave
    Jul 1, 2022 at 15:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.