Suppose I'd love to access the effect size and significance between outcome Y and variable X adjusted by confounder Z.

My question is that if there is any difference to determine the effect size and significance of X between following scenario.

  1. put variable and confounder together in a linear regression model. This means just fit regression model of Y ~ X + Z, then calculate the coefficient and its p-value of X.
  2. Get the residual, R from Y ~ Z, and then fit regression model of R ~ X, then calculate the coefficient and its p-value of X (from R~X).

I learn the confounder from here.

Edit -----

I appreciate @Gordon Smyth's answer. However, from a simulation study (code below), where I compared the false discovery rate of method1, method2, and the method3 from Gordon Smyth's answer, I surprisingly found that the method2 has a pretty low false positive rate.

I understand that method 1 is "textbook" correct. I wonder what exactly is wrong with the method2 logically? Besides, "All models are wrong, but some are useful".

p1 = p2 = p3 = c()
  y = rnorm(10)
  x = rnorm(10)
  c = rnorm(10)

  # method 1
  p1[i] = summary(lm(y~x + c))$coefficients[2,4]
  # method 2
  p2[i] = summary(lm(lm(y ~ c)$res ~ x))$coefficients[2,4]
  # method 3
  p3[i] = summary(lm(lm(y ~ c)$res~lm(x ~ c)$res))$coefficients[2,4]

  i = i+1

# number of false positive.
sum(p1<0.05) # 484
sum(p2<0.05) # 450
sum(p3<0.05) # 623
  • 1
    $\begingroup$ I'm not clear why you are surprised that the three methods are doing what I told you they would do in my answer a year ago. I told you that method 2 would be conservative (which is what you have found) and that method 3 would be anti-conservative (which again is what you have found). Anyway, the real problem with your method 2 only appears when X and Z are correlated with each other as well as with Y. I have expanded my answer now to explain this is more detail. I now give a numerical example showing how seriously bad method 2 can be. $\endgroup$ – Gordon Smyth May 25 '18 at 8:56

You need to adjust X as well as Y for the confounder

The first approach (using multiple regression) is always correct. Your second approach is not correct as you have stated it, but can be made nearly correct with a slight change. To make the second approach right, you need to regress both $Y$ and $X$ separately on $Z$. I like to write $Y.Z$ for the residuals from the regression of $Y$ on $Z$ and $X.Z$ for the residuals from the regression of $X$ and $Z$. We can interpret $Y.Z$ as $Y$ adjusted for $Z$ (same as your $R$) and $X.Z$ as $X$ adjusted for $Z$. You can then regress $Y.Z$ on $X.Z$.

With this change, the two approaches will give the same regression coefficient and the same residuals. However the second approach will still incorrectly compute the residual degrees of freedom as $n-1$ instead of $n-2$ (where $n$ is the number of data values for each variable). As a result, the test statistic for $X$ from the second approach will be slightly too large and the p-value will be slightly too small. If the number of observations $n$ is large, then the two approaches will converge and this difference won't matter.

It's easy to see why the residual degrees of freedom from the second approach won't be quite right. Both approaches regress $Y$ on both $X$ and $Z$. The first approach does it in one step while the second approach does it in two steps. However the second approach "forgets" that $Y.Z$ resulted from a regression on $Z$ and so neglects to subtract the degree of freedom for this variable.

The added variable plot

Sanford Weisberg (Applied Linear Regression, 1985) used to recommend plotting $Y.Z$ vs $X.Z$ in a scatterplot. This was called an added variable plot, and it gave an effective visual representation of the relationship between $Y$ and $X$ after adjusting for $Z$.

If you don't adjust X then you under-estimate the regression coefficient

The second approach as you originally stated it, regressing $Y.Z$ on $X$, is too conservative. It will understate the significance of the relationship between $Y$ and $X$ adjusting for $Z$ because it underestimates the size of the regression coefficient. This occurs because you are regressing $Y.Z$ on the whole of $X$ instead of just on the part of $X$ that is independent to $Z$. In the standard formula for the regression coefficient in simple linear regression, the numerator (covariance of $Y.Z$ with $X$) will be correct but the denominator (the variance of $X$) will be too large. The correct covariate $X.Z$ always has a smaller variance than does $X$.

To make this precise, your Method 2 will under-estimate the partial regression coefficient for $X$ by a factor of $1-r^2$ where $r$ is the Pearson correlation coefficient between $X$ and $Z$.

A numerical example

Here is a small numerical example to show that the added variable method represents the regression coefficient of $Y$ on $X$ correctly whereas your second approach (Method 2) can be arbitrarily wrong.

First we simulate $X$, $Z$ and $Y$:

> set.seed(20180525)
> Z <- 10*rnorm(10)
> X <- Z+rnorm(10)
> Y <- X+Z

Here $Y=X+Z$ so the true regression coefficients for $X$ and $Z$ are both 1 and the intercept is 0.

Then we form the two residual vectors $R$ (same as my $Y.Z$) and $X.Z$:

> R <- Y.Z <- residuals(lm(Y~Z))
> X.Z <- residuals(lm(X~Z))

The full multiple regression with both $X$ and $Y$ as predictors gives the true regression coefficients exactly:

> coef(lm(Y~X+Z))
(Intercept)           X           Z 
   5.62e-16    1.00e+00    1.00e+00 

The added variable approach (Method 3) also gives the coefficient for $X$ exactly correct:

> coef(lm(R~X.Z))
(Intercept)         X.Z 
  -6.14e-17    1.00e+00 

By contrast, your Method 2 finds the regression coefficient to be only 0.01:

> coef(lm(R~X))
(Intercept)           X 
    0.00121     0.01170 

So your Method 2 underestimates the true effect size by 99%. The under-estimation factor is given by the correlation between $X$ and $Z$:

> 1-cor(X,Z)^2
[1] 0.0117

To see all this visually, the added variable plot of $R$ vs $X.Z$ shows a perfect linear relationship with unit slope, representing the true marginal relationship between $Y$ and $X$:

Added variable plot

By contrast, the plot of $R$ vs the unadjusted $X$ shows no relationship at all. The true relationship has been entirely lost:

Incorrect plot using uncorrected X

  • 1
    $\begingroup$ Are You sure that both approaches will give same regression coefficients? If it was true then instead of doing multiple regression with k independent variables we could do k regressions with single independent variable and obtain exactly the same result way faster. $\endgroup$ – Tomek Tarczynski Jun 28 '17 at 10:47
  • 1
    $\begingroup$ I'm probably missing something. When I try to simulate this I'm getting different coefficient for X. R code is below set.seed(1234) k <- 100 x <- runif(k) z <- x + runif(k) y <- 5*x - 3*z + runif(k) # x coefficient 5.1252 lm(y ~ x + z) model <- lm(y ~ z) res <- model$residuals #x coefficient 2.82 model2 <- lm(res ~ x) $\endgroup$ – Tomek Tarczynski Jun 29 '17 at 7:22
  • 1
    $\begingroup$ @TomekTarczynski You're right, I forgot to point out that $X$ must be regressed on $Z$ as well. I have edited my answer to correct it. $\endgroup$ – Gordon Smyth Jun 29 '17 at 8:47
  • $\begingroup$ Hi Gordon, is there is anything wrong with the second approach other than the issue of the degree of freedom? e.g. In terms of logics? $\endgroup$ – WCMC Dec 15 '17 at 0:12
  • $\begingroup$ Hi Gordon, I edited my question. I appreciate if you could have a look when you have time. $\endgroup$ – WCMC May 24 '18 at 5:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.