# Using control variables in experiments?

Why would one want to control for any number of baseline covariates in a situation where the assignment to treatment group is random?

My understanding is that randomly assigning treatment should make the treatment variable strictly exogenous, creating a control group that can appropriately be considered as a counterfactual. The only exception I can think of is when sample sizes are small, and that random assignment can still produce unbalanced groups.

Any thoughts are much appreciated. Thanks!

-

From a frequentist perspective, an unadjusted comparison based on the permutation distribution can always be justified following a (properly) randomized study. A similar justification can be made for inference based on common parametric distributions (e.g., the $t$ distribution or $F$ distribution) due to their similarity to the permutation distribution. In fact, adjusting for covariates—when they are selected based on post-hoc analyses—actually risks inflating the Type I error. Note that this justification has nothing to do with the degree of balance in the observed sample, or with the size of the sample (except that for small samples the permutation distribution will be more discrete, and less well approximated by the $t$ or $F$ distributions).

That said, many people are aware that adjusting for covariates can increase precision in the linear model. Specifically, adjusting for covariates increases the precision of the estimated treatment effect when they are predictive of the outcome and not correlated with the treatment variable (as is true in the case of a randomized study). What is less well known, however, is that this does not automatically carry over to non-linear models. For example, Robinson and Jewell [1] show that in the case of logistic regression, controlling for covariates reduces the precision of the estimated treatment effect, even when they are predictive of the outcome. However, because the estimated treatment effect is also larger in the adjusted model, controlling for covariates predictive of the outcome does increase efficiency when testing the null hypothesis of no treatment effect following a randomized study.

[1] L. D. Robinson and N. P. Jewell. Some surprising results about covariate adjustment in logistic regression models. International Statistical Review, 58(2):227–40, 1991.

-
Hello - interesting answer. Any interest in having an offline conversation about this? –  rolando2 Jan 25 '14 at 17:51

If the outcome depends on treatment as well as other observable factors, controlling for the latter often improves the precision of the impact estimate (i.e., the standard error of the treatment effect will be smaller). When sample size is small, this can be helpful.

Here's a simple simulation where even though treatment is random, the standard error shrinks by a third:

. set obs 100
obs was 0, now 100

. gen treat =mod(_n,2)

. gen x=rnormal()

. gen y = 2 + 3*treat + 1*x + rnormal()

. reg y treat

Source |       SS       df       MS              Number of obs =     100
-------------+------------------------------           F(  1,    98) =  112.75
Model |  209.354021     1  209.354021           Prob > F      =  0.0000
Residual |  181.973854    98  1.85687606           R-squared     =  0.5350
-------------+------------------------------           Adj R-squared =  0.5302
Total |  391.327875    99  3.95280682           Root MSE      =  1.3627

------------------------------------------------------------------------------
y |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
treat |   2.893814   .2725345    10.62   0.000     2.352978     3.43465
_cons |   2.051611    .192711    10.65   0.000     1.669183     2.43404
------------------------------------------------------------------------------

. reg y treat x

Source |       SS       df       MS              Number of obs =     100
-------------+------------------------------           F(  2,    97) =  180.50
Model |  308.447668     2  154.223834           Prob > F      =  0.0000
Residual |  82.8802074    97  .854435127           R-squared     =  0.7882
-------------+------------------------------           Adj R-squared =  0.7838
Total |  391.327875    99  3.95280682           Root MSE      =  .92436

------------------------------------------------------------------------------
y |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
treat |   2.918349   .1848854    15.78   0.000     2.551403    3.285295
x |   1.058636   .0983022    10.77   0.000     .8635335    1.253739
_cons |   1.996209    .130825    15.26   0.000     1.736558     2.25586
------------------------------------------------------------------------------

-
+1 - Another reason is to identify interaction effects with treatment, but that takes more than just "controlling" for other factors like the OP mentioned. –  Andy W Jan 24 '14 at 23:55
Thanks! So, if a one or more covariates affect the outcome you're trying to measure, including them in your model will improve the precision of your estimate of the randomly assigned treatment effect, but won't really impact your estimate of the value of the treatment coefficient, correct? –  Robb Jan 26 '14 at 2:57
Yes, that's right. –  Dimitriy V. Masterov Jan 26 '14 at 18:01