2
$\begingroup$

It is a very similar question to this one.

In my case, the exact question was why adding a new variable in a linear regression model is the same as controlling for it. I am going to use a similar R simulation as @Fomite and @Bernd Weiss in their answers to the original question, with some changes (now the covariate will be continue, sampled from a normal distribution, and I added some noise). I wonder if what I am doing is correct or not (but the results look consistent).

Controlling for a variable should mean (as I understand it) that we are conditioning on its values. That is, we are doing the regression for fixed values of it (this is how I intuitively understand the term). So, another way (very cumbersome) to do it could be running several (in the limit, if the covariate is continue, infinite) regressions, one for each value of the covariate. And we could combine all coefficients into a single one by doing a weighted average.

I have this simulation, and what I get looks correct.

exposure  <- runif(100000,0,1)+(0.3*covariate)+rnorm(100000, 0, 0.1) 
outcome   <- 2.0+(0.5*exposure)+(0.25*covariate)+rnorm(100000, 0, 0.1) 

lm(outcome~exposure)


lm(outcome~exposure+covariate)


n.bins <- 100
coefs <- rep(0,n.bins)
nums <- rep(0,n.bins)
for (bin in 0:(n.bins-1)){
  cov.b <- covariate>bin/n.bins & covariate<(bin+1)/n.bins 
  linear.mod <- lm(outcome[cov.b]~exposure[cov.b])
  coefs[bin+1] <- linear.mod$coefficients[2]
  nums[bin+1] <- sum(cov.b)
}
coefs
final.coef <- sum(coefs * nums) / sum(nums)
final.coef

That is, both in the linear model with the covariate and in the final weighted coefficient (created by averaging all coefficients for given values of the covariate) I get a similar result (0.5, within a small error, which is the true coefficient).

If the reasoning is correct, a formula like this should exist (but I've not found it anywhere):

$$ y = \beta_0 + \beta_1 \cdot x_1 + \beta_2 \cdot x_2 $$

$$ \beta_2 = \mathbb{E}_{x_1}(\hat{\beta_2} | X_1 = x_1) $$

I am not sure if I am being precise enough with the notation. I mean with the previous formula that I am taking the expectation (over the distribution of $X_1$) of the coefficient of linear regression of $y = \beta_0 + \beta_2 \cdot x_2$ using only the data points $X_2$ and $y$ for which $X_1$ takes the value of $x_1$.

Does it make sense?

EDIT

The simulation is more interesting, I think, when the true data generating process is not linear and contains interactions. For example:

covariate <- rnorm(100000,0.5,0.2)
exposure  <- runif(100000,0,1)+(0.3*covariate)+rnorm(100000, 0, 0.1) 
outcome   <- 2.0+(2*exposure)+(1.5*covariate) +(3*covariate*covariate)+ 2*covariate*exposure+rnorm(100000, 0, 0.1) 

lm(outcome~exposure)

lm(outcome~exposure+covariate)

n.bins <- 100
coefs <- rep(0,n.bins)
nums <- rep(0,n.bins)
for (bin in 0:(n.bins-1)){
  cov.b <- covariate>bin/n.bins & covariate<(bin+1)/n.bins 
  linear.mod <- lm(outcome[cov.b]~exposure[cov.b])
  coefs[bin+1] <- linear.mod$coefficients[2]
  nums[bin+1] <- sum(cov.b)
}
coefs
final.coef <- sum(coefs * nums) / sum(nums)
final.coef

In this case, the slopes of outcome vs exposure change for each value of the covariate, so the different coefficients that I get in the simulation vary accross the levels of the covariate. But the weighted average is pretty, pretty close to the one given by the linear model with both terms.

lm(formula = outcome ~ exposure + covariate)

Coefficients:
(Intercept)     exposure    covariate  
     0.7416       3.0021       5.8026  



coefs
  [1] 2.027351 1.993495 2.055792 2.093611 2.041095 2.047883 2.120171 2.148726 2.163537 2.189357 2.185651 2.226658 2.281700 2.241693 2.301020 2.286992
 [17] 2.327823 2.343747 2.382957 2.390632 2.420379 2.438041 2.443451 2.460140 2.478369 2.519187 2.506563 2.563371 2.574275 2.569280 2.610162 2.616787
 [33] 2.668466 2.661996 2.683032 2.703801 2.730615 2.751894 2.762410 2.797779 2.822186 2.841390 2.829124 2.864518 2.890004 2.907125 2.928860 2.958108
 [49] 2.976774 2.984160 2.988439 3.028318 3.047874 3.062519 3.110787 3.114223 3.109860 3.151624 3.184492 3.188091 3.210913 3.239821 3.249189 3.272366
 [65] 3.299281 3.318268 3.324639 3.359445 3.372609 3.394405 3.411057 3.445723 3.460289 3.484919 3.493236 3.507415 3.538879 3.562681 3.576665 3.597904
 [81] 3.607418 3.622272 3.621013 3.702007 3.670223 3.714877 3.757372 3.735650 3.786936 3.809358 3.799560 3.821558 3.825680 3.807918 3.852794 3.895477
 [97] 3.939175 3.944392 4.031231 4.022685


final.coef
[1] 2.998887

So I feel that the reasoning is right... But I'd like to have a confirmation.

EDIT 2:

I'm trying to formalize better the conjecture. Let's think about a linear regression with two features and one outcome:

$$ y = \beta_0 + \beta_1 \cdot x_1 + \beta_2 \cdot x_2 $$

Then, we can think of conditioning on $X_1$ as having a one-parametric family of probability distributions, one for each value $x_1$ of $X_1$, with density $P(x_1)$.

So I'd say that, if we fit a linear regression for each value $x_1$ of $X_1$:

$$ y | x_1 = \beta_0 (x_1) + \beta_2 (x_1) \cdot x_2 | x_1 $$

This will give us a distribution of $\beta$ coefficients: $$ P(\beta_2 (x_1)) = P(x_1)$$ If we take the expectation of the previous distribution, we will get the $\beta_2$ coefficient of the full linear model that contains the two variables.

If the previous reasoning is true, the relationship between conditioning, controlling and adding a variable to a linear model will be much clearer (at least for me).

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.