2
$\begingroup$

Suppose we have an outcome $Y$ which is caused by a binary treatment $T$ and some continuous covariate $X$, such that:

$$E[Y\mid X,T=1] = E[Y^1\mid X] = \beta_0 + \delta + \beta_X X$$

$$E[Y\mid X,T=0] = E[Y^0\mid X] = \beta_0 + \beta_X X$$

where $\beta_0$ is an intercept and $\delta$ is the average treatment effect ($ATE$).

Suppose also that we have a sample of two units:

A control unit ($T=0$) with $X=5$

A treated unit ($T=1$) with $X=5.5$

Ignoring for simplicity any noise or sampling variance, if we were to estimate the $ATE$ as the difference between the outcomes of those two units, our estimate $\widehat{ATE}$ would be biased:

enter image description here

Here the true $ATE$ is the vertical distance between the parallel regression planes, but our $\widehat{ATE}$ is overestimating. Importantly, this bias is present whether or not $X$ is a confounder (i.e. whether or not $X$ also has an effect on $T$).

What I just noticed is that this bias becomes bigger if the effect of $X$ on $Y$ is stronger (i.e. if the regression planes are steeper):

enter image description here

In the language of matching, it seems that the biasing effect of having an imperfect match is stronger when the slope of $X$ is larger, even if $X$ is not a confounder.

I realize this bias will vanish in larger samples because the mismatches will average out, but I think it will vanish slower if the slopes are steep and, at least in my work, it could be confused for confounding bias in small samples.

Is this a thing?

$\endgroup$

2 Answers 2

2
$\begingroup$

I realize this bias will vanish in larger samples because the mismatches will average out, but I think it will vanish slower if the slopes are steep and, at least in my work, it could be confused for confounding bias in small samples.

If $X$ and $T$ are independent, then there is no bias, for any sample size: $$E[Y|T=1] - E[Y|T=0] = \delta + \beta_x (E[X|T=1] - E[X|T=0]) = \delta$$

The effect you're describing does make the difference in treated and control units a worse estimator of $\delta$, however, because it increases its variance. Randomly, if not systematically, the $X$ values of treated units will differ from the $X$ value of control units, and that difference will end up causing a difference in $Y$ as well. This increases the variability of treatment and control units and so also the variability of their difference:

$$Var(Y|T=1) = Var(Y|T=0) = \beta_x ^2 Var(X)$$

As you can see, a larger $\beta_x$ will translate into more variability in the difference of outcomes that isn't due to the treatment effect $\delta$, making it a worse estimator.

This is why it's often recommend that you add $X$ to the model even when you run an experiment which guarantees $T$ is not confounded by $X$. Controlling for $X$ will reduce the variance and gives a more precise estimator. The stronger $X$ predicts $Y$ ($\beta_x$ larger), the larger the gains in precision if you include $X$ in the model.

$\endgroup$
1
  • $\begingroup$ Excellent explanation, couldn't have done better myself. OP is comparing the outcomes of the two units, not conditioning on X. The sample is imbalanced, so the effect estimate will be estimated with error. $\endgroup$
    – Noah
    Aug 7, 2019 at 20:00
1
$\begingroup$

This is not an issue of small sample bias. Small sample bias refers to an interesting mathematical anomale of binary outcomes when the sampling distribution of the estimates are highly discretized.

http://web.pdx.edu/~newsomj/cdaclass/ho_sample%20size.pdf

In your sample of 2 observations, the model of 3 parameters -- the intercept, the $\beta_x$ effect, and the $\delta$ effect -- is oversaturated. The model is not even estimable. Add another observation and the estimates are, on average, correct. Whether $X$ should or should not be adjusted for has nothing to do with the sample size.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.