When is there a point for using regression with controls to analyze experimental data?

Question

Many randomized controlled trials in economics (and probably other sciences as well?) use regressions to analyze data obtained from experiments. In the simplest setup with binary treatment, assuming a constant treatment effect, a typical regression equation would look like:

$$y_i = \alpha + \beta T_i + Z_i\theta_Z + \varepsilon_i,$$

where $T_i$ is a dummy indication treatment vs. placebo and $Z_i$ is a vector of control variables holding all kinds of information, which the researcher expects to predict the outcome. By design (randomization) $T_i$ is uncorrelated with $Z_i$. The treatment effect estimate is then simply the coefficient estimate of $\beta$, $\hat\beta$.

How can the use of this be justified above a simple test of difference in means between the two groups? This would correspond to estimating $\beta$ in a regression without controls: $$y_i = \alpha + \beta T_i + \varepsilon_i$$

Is it increasing the precision of the estimate?

A research is interested in getting the most precise estimate for the average treatment effect. I.e. (s)he might be using controls to improve/lower $V(\hat\beta)$. When does that work?

Let $\theta = (\beta,\alpha,\theta_Z')'$, $X_i = (T_i, 1,Z_i')$, and let $X$ be a matrix stacking all the $X_i$. Then a regression equation in matrix notation with controls would be: $$Y=X\theta+\varepsilon$$ Now, the variance of the OLS estimate for $\theta$ is $\hat\theta=(X'X)^{-1}X'Y$. The variance of this decomposes to: $$V(\hat\theta)=E[(X'X)^{-1}X'V[\varepsilon|X]X(X'X)^{-1}]$$ If we assume homoscedasticity $V[\varepsilon|X]=\sigma^2I$, this becomes: $$V(\hat\theta)=\sigma^2E[(X'X)^{-1}]$$

The variance of my treatment effect estimate, $V(\hat\beta)$ is now the first element in $V(\hat\theta)$

Now I guess one possible goal would be to minimize this (and thereby minimize MSE). But it is not obvious to me when this would lead to the use of covariates $Z_i$. More precisely, the use of covariates could do both, increase and decrease the variance of $\hat\beta$. Is it possible to derive conditions for this?

Answer 1

An alternative way to write the standard error (assuming homoscedasticity) of the regression coefficient estimate in your first model is \begin{equation} se(\hat \beta) = \sqrt{\frac{\sum_{i=1}^n e^2}{(n-k-1)\sum_{i=1}^n(T_{i}-\bar T)^2(1-R^2_T)}} \end{equation} where $e$ are the residuals from the regression and $R^2_T$ is the $R^2$ from a regression of $T$ on $Z$. Three things in the equation could change depending on if you include $Z$ in the model or not.

First, the sum of squared residuals will always decrease (technically never increases) since we will never explain less of $y$ by adding regressors. This will make the standard error smaller.

Second $k$, the number of regressors, will change. This increases the standard error.

Third, $R^2_T$ will likely not be too different from 0 since random assignment ensures that $Z$ is uncorrelated with $T$. I will assume this is 0, but larger values will increase the standard error.

From the first two adding more regressors will decrease the standard error as long as the decrease in sum of squared residuals (unexplained part of $y$) is large relative to the increase in the number of coefficients.