4
$\begingroup$

After conducting the experiment and doing the statistical analysis, usually we run a linear regression $Y_i=\alpha+\beta X_i+\epsilon_i$, where $X_i$ is the treatment variable (randomly assign people to treatment or control group). Often times, the researcher may find that there is another attributes observed on subjects, say gender, then the researcher may also incorporate this variable into the regression, i.e., $Y_i=\alpha'+\beta' X_i+\gamma' Z+\epsilon'_i$, where $Z$ is the attributes of subjects (e.g., gender).

My question is:

(1) if the randomization procedure is effective, then in 2 groups, the distribution of the attribute $Z$ should roughly the same right? Then what do we gain from the statistical analysis by incorporating the attributes variable $Z$ into the regression?

(2) before adding $Z$, we can show that $E[\epsilon_i|X_i]=0$ and $\beta=E[Y_i|X_i=1]-E[Y_i|X_i=0]$ and by randomization, we know that $E[Y_i|X_i=1]-E[Y_i|X_i=0]=E[Y_{1i}-Y_{0i}]$, where $Y_{1i}$ means the distribution of the response variable in the treatment group and $Y_{0i}$ is for the control group. Now if we add $Z$ into the regression, can we prove in the new regression $E[\epsilon'_i|X_i,Z_i]=0$? Also, will $\beta'=\beta$?

I couldn't understand this at an intuitive level. However, I made the following effort to understand this problem. Here is what I did:

Let $Z$ also be a dummy variable, takes value 0 or 1, then we have 4 combinations of $(X,Z)$. I generate 4 populations using R

Y11 = rnorm(N,m11,s11),  Y10 = rnorm(N,m10,s10)

Y01 = rnorm(N,m01,s01),  Y00 = rnorm(N,m00,s00) 

where m01,m11,m10,m00, s11,s01,s10,s00 are different numbers. Then I can compute the mean of these 4 populations

M11 = mean(Y11), M10 = mean(Y10), M01 = mean(Y01), M00 = mean(Y00)

Then I generate random variable $Z$ (say, the gender of each individual, I know it is not a good example) such that I explicitly made $Z$ is correlated with some of these $Y_{ab}$'s. Specifically, I did

Z = (Y11-M11>0)&(Y01-M01>0)

Then I create two mixture distribution

Mix1 = ifelse(Z>0,Y11,Y10)

Mix2 = ifelse(Z>0,Y01,Y00)

One can think of the distribution of Mix1 as: you have a population, where there are man and woman, and for some reason some individuals are men and others are women, then you force all of them to take treatment A, then you observe the distribution of the response variable, which gives you the distribution of Mix1. So for Mix2, it is like that you made a copy of the population I just described, then you force all of them to take treatment B, then you observe another distribution of the response variables, which gives you the distribution of Mix2.

Then I generate a complete list of independent Bernoulli random variables to represent the randomization procedure did in the experiment.

D = rbinom(N,1,0.4) # with 0.4 probability one guy is assigned to Treatment A

Then I created the "observed data set" from experiment

data = ifelse(D>0,Mix1,Mix2)

Now, I run two regression, one with $Z$ and one without $Z$, I have

Res1 = lm(data~D+Z)

Res2 = lm(data~D)

The results are that $\hat{\beta}\approx\hat{\beta'}\approx$mean(Mix1-Mix2), such that I could claim that $\beta=\beta'=$mean(Mix1-Mix2). And I found that $\hat{\beta}'$, which comes from the regression with $Z$ included, is closer to the truth, mean(Mix1-Mix2), and it has less standard error smaller than $\hat{\beta}$.

So this R exercise seems to suggest that as long as you perform randomization to the treatment variable, then when you do the analysis, you could add whatever attributes variable you observed on the subjects, and just put them into the regression and this will help you with finding the true counterfactual mean treatment effect than you just run a simple linear regression with the treatment variable only. (Of course you cannot add an interaction term, since in that case, the meaning of the coefficient of $X$ changes)

At intuitive level, why it is true? And can someone provide a mathematical proof of that? I know how to prove it when we don't have $Z$ in the regression.

$\endgroup$
3
$\begingroup$

If randomization was successful, then potential outcomes are independent of the treatment unconditionally but also conditionally of covariates. That is $$ (Y_{i0}, Y_{i1}, Z_i)\perp \hspace{-0.2cm}\perp X_i \quad \text{which implies} \quad (Y_{i0}, Y_{i1}) \perp \hspace{-0.2cm}\perp X_i \mid Z_i $$ For more information on this have a look at these lecture notes by Clément de Chaisemartin, section 5.2 on page 79.

Using the Frisch-Waugh theorem and the omitted variable bias formula, you know that your estimated coefficient on $X_i$ is $$ \widehat{\beta} = \beta + \gamma \frac{Cov(X_i,Z_i)}{Var(X_i)} $$

and since $X_i$ and $Z_i$ are independent, $Cov(X_i,Z_i) = 0$, and therefore your coefficient estimate is unchanged whether or not you include/exclude $Z_i$.
In your simulation exercise you are making one error: you should do the simulation repeatedly and see what happens to your $\widehat{\beta}$ and $\widehat{\beta}'$. Once you do this, say, a hundred times you will find that both estimates converge on the true population value.

However, in randomized experiments it has a big advantage to include other covariates: it reduces the standard errors and makes inference on your treatment more precise. This is because additional covariates soak up part of the residual variance which reduces the overall estimation error (that's the intuitive explanation).
The mathematical proof for this is provided in this answer.

$\endgroup$
  • $\begingroup$ is the notation $(Y_{i0},Y_{it}) || X_i|Z_i$ means that given the value of $Z_i$, then $(Y_{i0},Y_{1i})$ is independent of $X_i$? $\endgroup$ – KevinKim Sep 2 '16 at 18:12
  • $\begingroup$ Yes but given that $X_i$ is also independent of $Z_i$ (see the expression on the left) it doesn't matter whether you condition on $Z_i$ or not. $\endgroup$ – Andy Sep 2 '16 at 18:26
  • $\begingroup$ What if Z is also a randomized treatment, say, we have a 2 by 2 treatment, then I run the regression $Y= \alpha+\beta X+\gamma Z+\epsilon$, then is this the same as the situation we are talking about now? $\endgroup$ – KevinKim Sep 2 '16 at 18:51
  • $\begingroup$ Yes, because again you will have that $X_i$ and $Z_i$ are independent of each other, meaning that $Cov(X_i,Z_i)=0$. Therefore it makes no difference to the coefficient of one treatment variable when you leave out the other treatment. $\endgroup$ – Andy Sep 2 '16 at 18:55
  • $\begingroup$ Then what if I first do "blocking", i.e., I put all people with Z=1 in one group, and then all people with Z=0 in another group, then within each group, I do randomization on X. Finally, I still run the regression $Y=\alpha+\beta X+\gamma Z+\epsilon$. In this case, is it the same as the previous 2 cases? Here Z cannot be viewed as a random variable, so then is $Cov(X,Z)$ still meaningful? $\endgroup$ – KevinKim Sep 2 '16 at 19:08
1
$\begingroup$

If X and Z are completely independent, then $\beta'$=$\beta$ and adding Z will increase the $R^2$ or decrease SE.

In OLS, if any of the explanatory variables are independent, then the coefficients on them are identical to a regression leaving out the other variable. (not difficult to prove at all if you want a proof).

If X and Z are not independent, then you want to include Z if it is related to Y. So in your case, not including Z makes your treatment have selection bias, so $E(e|X)\not=0$

However if we add Z, it may be the case that now conditional on this new variable, your treatment is random: $E(e|X, Z)=0$ , so conditional on the added variable, your treatment is now valid.

This is known as 'selection on observables' or 'conditional independence' or 'unconfounded assignments'. Search these terms or read any econometrics/statistical inference book like Imbens and Rubin for more.

$\endgroup$
  • $\begingroup$ I think in my case, X and Z are independent right? since whether a person is assigned to treatment A or B is via randomization. Hence, if I know one person is assigned to treatment A, this does not help me to better estimate the distribution of Z, is that right? $\endgroup$ – KevinKim Sep 2 '16 at 17:54
  • $\begingroup$ So to estimate the coefficient, including Z does nothing. To get smaller SEs or increase R2, then adding Z can help. $\endgroup$ – VCG Sep 2 '16 at 17:58
  • $\begingroup$ then is it true that if the treatment is completed randomized, then when we do statistical analysis, it's good to add those attributes (of individual) variables into the regression, since these variables are independent of the treatment variable and by incorporating them, we can estimate the coefficient, the counterfactual effect more accurately? $\endgroup$ – KevinKim Sep 2 '16 at 18:03
  • $\begingroup$ The coefficient on treatment will be unaffected by those new variables, but the fit/precision of the overall model may be improved. $\endgroup$ – VCG Sep 2 '16 at 18:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.