# Difference in estimating treatment effect in experiment using regression with covariates and assuming the data come from stratified experiment?

Assume I only did a completely randomized experiment (CRE), i.e., $N$ subjects, I randomly assign $N_t$ (say $N_t=N/2$) of them to the treatment group and the rest to the control group. Now, after I finish the experiment and try to compute the average treatment effect. I also see that I recorded the gender of each subject. Now assume my sample size is large, i.e., $N$ is big, then as the randomization is successful, I find that within each group, the gender is well-balanced, i.e., it "looks like" I did a blocking first and randomized second (stratified randomized experiment (SRE)). Now my question is the following:

Let $Y_i(0)$ be the potential outcome of subject $i$ if he is assigned to control group and $Y_i(1)$ be the potential outcome of subject $i$ if he is assigned to treatment group. Our goal is to get $\tau_{fs}=\frac{1}{N}\sum_{i=1}^N[Y_i(1)-Y_i(0)]$. This is impossible to directly get, as for each subject, we only observe one of his potential outcome. Now here I have 2 estimator to estimate this quantity $\tau_{fs}$. Let $W_i\in\{0,1\}$ denote whether subject $i$ is assigned to control ($W_i=0$) or treatment group ($W_i=1$) and let $X_i\in\{m,f\}$ be the subject $i$'s gender. Let $Y^{obs}_i=Y_i(W_i)$ be the observed value of subject $i$. Let $N_t$ and $N_c$ be the total number of people in the treatment and control group

\begin{equation} \hat{\tau}_1=\frac{1}{N_t}\sum_{i:W_i=1}Y^{obs}_i-\frac{1}{N_c}\sum_{i:W_i=0}Y^{obs}_i \end{equation} This is just take the average within each group, then take the difference. Here is a more complex one, let $N_f$ and $N_m$ be the total number of female and male in the sample. \begin{equation} \hat{\tau}_2=\frac{N_f}{N}\hat{\tau}_f+\frac{N_m}{N}\hat{\tau}_m \end{equation} where \begin{equation} \hat{\tau}_x=\frac{1}{N_t(x)}\sum_{i:X_i=x,W_i=1}Y^{obs}_i-\frac{1}{N_c(x)}\sum_{i:X_i=x,W_i=0}Y^{obs}_i,~\mbox{with }x\in\{m,f\} \end{equation} Namely, this estimator, "pretend" that we did a stratified random experiment, so it first take the treatment effect within each category based on the gender, then it did a weighted average of these two treatment effects and weight is the fraction of male (and female) in the sample. One can show that both $\hat{\tau}_1$ and $\hat{\tau}_2$ are unbiased estimator of $\tau_{fs}$.

I did a simulation study, I found that $\hat{\tau}_2$ seems always better than $\hat{\tau}_1$, I wonder if there is any mathematical or intuitive justification for that? Also, one can show that the variance of $\hat{\tau}_2$ is less than $\hat{\tau}_1$. So in practice is there a reason NOT to use $\hat{\tau}_2$ but just use $\hat{\tau}_1$? Will this weighed average $\hat{\tau}_2$ lead to something like Simpson's paradox?

As stratified random experiment requires extra work than completely randomized experiment, in practice, IF $N$ IS BIG, should we just do a completely randomized experiment and in the data analysis part, we just use $\hat{\tau}_2$ and its variance, i.e., "pretend" the data comes from stratified random experiment? My justification is that as $N$ is big, so the attributes will be balanced within each group (and one can check it empirically). I understand that when $N$ is small, then just a completely randomized experiment without stratified would lead you to the situation like all male in the treatment group and all female in the control group, then the result is useless. So when $N$ is small, then stratified random experiment eliminates this possibility. But other than that, I don't see a point of doing stratified random experiment, especially considering that it requires extra work than just completely randomized experiment. Did I miss something important?

The $\hat{\tau}_2$ estimator has lower variance than $\hat{\tau}_1$. However, when analyzing the data, we could also run a linear regression with the treatment variable and the gender attributes variable, then we can derive the ols estimator for the treatment effect (the coefficient in front of the treatment variable) as $\hat{\tau}_3$. Note that the variance of $\hat{\tau}_3$ is also lower than that of $\hat{\tau}_1$. But $\hat{\tau}_3$ and $\hat{\tau}_2$ are equal. Which one should we use?

Specifically, one can run the following regression model \begin{equation} Y_i = \alpha' + \tau'_3W_i+\epsilon' \end{equation} It can be shown that the the ols estimator of the above model denoted by $\hat{\tau}'_3=\hat{\tau}_1$.

One can also run another regression model with covariates (gender) included \begin{equation} Y_i = \alpha + \tau_3W_i+\beta X_i+\epsilon \end{equation} It can be shown that the ols estimator $\hat{\tau}_3$ is consistent for the true treatment effect and $\hat{\tau}_3$ has less variance than $\hat{\tau}_1$.

In addition, for $\hat{\tau}_2$, it in the form of weighted average. But I also see some people use different weight than the gender proportion. Instead, they use inverse of variance within each group as the weight. But in that case will $\hat{\tau}_2$ still be a unbiased estimator of $\tau_{fs}$?

• You seem to know the concept of potential outcomes. If your treatment and control groups are exchangeable, then you need not do any fancy math. Your second estimator looks like an inverse probability weighted estimator, basically. Say your two groups are balanced. I can't see any reason to use IPW. You would use IPW in an observational study, or if your randomization had failed. I also can't see any reason it should have lower variance than the first estimator. Explain your simulation study a bit more? – Weiwen Ng May 24 '17 at 22:21
• @WeiwenNg I am reading the Rubin's Causal Inference book. A great book. The 2nd estimator has less variance than the 1st one. So I think we should use the 2nd one, especially say you want to have a confidence interval for the treatment effect.The 2nd estimator is used in the stratified experiment. But I think there is nothing wrong for me to use it in a non-stratified but random experiment. And when sample size is large and the randomization is successful, the gender distribution within treatment and control group should be balanced. I was wondering if there is a need to do any blocking first – KevinKim May 25 '17 at 0:50
• @WeiwenNg To me I guess the only reason that people do stratified experiment is to prevent unbalance distribution of gender (in this example) within treatment and control group from happening, which you may run into if you just do a completely randomized experiment. But when the sample size is large. Then I don't see an advantage of doing this stratified experiment. – KevinKim May 25 '17 at 0:54
• @WeiwenNg The simulation I did can be summarized as: generate $\{Y_i(0),Y_i(1),W_i,X_i\}_{i=1}^{N}$, where $W_i$ is a fair coin that is independent of the rest 3 r.v.s. Then $Y^{obs}_i=W_iY_i(1)+(1-W_i)Y_i(0)$. Then I can compute my 2 estimators and their variance. I also did 2 regressions, one without $X$ and one with $X$ included. $X$ is the gender variable. – KevinKim May 25 '17 at 0:58
• I am not an expert in this field but to add my two cents: it seems like there is an interaction effect of the outcome with gender? Maybe this doesn't help at all but just thought to mention it. – DataD'oh Aug 29 '17 at 16:24

I would have thought you are calculating conditional variance under the CRE though - this variance under CRE would be a function of the number of males/females selected in the sample. This is technically "random" under CRE and you should be using something like $$E \left[var \left(\hat{\tau}_2|N_t (m),N_c (m)\right)\right]+var\left [E\left(\hat{\tau}_2|N_t (m),N_c (m)\right)\right]$$ The second term depends on how like you are to get $0$ males/females in treatment/control groups for the CRE (and which estimate you would use if this happened). With stratification you eliminate the second term.