# Is this a well-known source of small-sample bias?

Suppose we have an outcome $$Y$$ which is caused by a binary treatment $$T$$ and some continuous covariate $$X$$, such that:

$$E[Y\mid X,T=1] = E[Y^1\mid X] = \beta_0 + \delta + \beta_X X$$

$$E[Y\mid X,T=0] = E[Y^0\mid X] = \beta_0 + \beta_X X$$

where $$\beta_0$$ is an intercept and $$\delta$$ is the average treatment effect ($$ATE$$).

Suppose also that we have a sample of two units:

A control unit ($$T=0$$) with $$X=5$$

A treated unit ($$T=1$$) with $$X=5.5$$

Ignoring for simplicity any noise or sampling variance, if we were to estimate the $$ATE$$ as the difference between the outcomes of those two units, our estimate $$\widehat{ATE}$$ would be biased: Here the true $$ATE$$ is the vertical distance between the parallel regression planes, but our $$\widehat{ATE}$$ is overestimating. Importantly, this bias is present whether or not $$X$$ is a confounder (i.e. whether or not $$X$$ also has an effect on $$T$$).

What I just noticed is that this bias becomes bigger if the effect of $$X$$ on $$Y$$ is stronger (i.e. if the regression planes are steeper): In the language of matching, it seems that the biasing effect of having an imperfect match is stronger when the slope of $$X$$ is larger, even if $$X$$ is not a confounder.

I realize this bias will vanish in larger samples because the mismatches will average out, but I think it will vanish slower if the slopes are steep and, at least in my work, it could be confused for confounding bias in small samples.

Is this a thing?

I realize this bias will vanish in larger samples because the mismatches will average out, but I think it will vanish slower if the slopes are steep and, at least in my work, it could be confused for confounding bias in small samples.

If $$X$$ and $$T$$ are independent, then there is no bias, for any sample size: $$E[Y|T=1] - E[Y|T=0] = \delta + \beta_x (E[X|T=1] - E[X|T=0]) = \delta$$

The effect you're describing does make the difference in treated and control units a worse estimator of $$\delta$$, however, because it increases its variance. Randomly, if not systematically, the $$X$$ values of treated units will differ from the $$X$$ value of control units, and that difference will end up causing a difference in $$Y$$ as well. This increases the variability of treatment and control units and so also the variability of their difference:

$$Var(Y|T=1) = Var(Y|T=0) = \beta_x ^2 Var(X)$$

As you can see, a larger $$\beta_x$$ will translate into more variability in the difference of outcomes that isn't due to the treatment effect $$\delta$$, making it a worse estimator.

This is why it's often recommend that you add $$X$$ to the model even when you run an experiment which guarantees $$T$$ is not confounded by $$X$$. Controlling for $$X$$ will reduce the variance and gives a more precise estimator. The stronger $$X$$ predicts $$Y$$ ($$\beta_x$$ larger), the larger the gains in precision if you include $$X$$ in the model.

• Excellent explanation, couldn't have done better myself. OP is comparing the outcomes of the two units, not conditioning on X. The sample is imbalanced, so the effect estimate will be estimated with error.
– Noah
Aug 7, 2019 at 20:00

This is not an issue of small sample bias. Small sample bias refers to an interesting mathematical anomale of binary outcomes when the sampling distribution of the estimates are highly discretized.

http://web.pdx.edu/~newsomj/cdaclass/ho_sample%20size.pdf

In your sample of 2 observations, the model of 3 parameters -- the intercept, the $$\beta_x$$ effect, and the $$\delta$$ effect -- is oversaturated. The model is not even estimable. Add another observation and the estimates are, on average, correct. Whether $$X$$ should or should not be adjusted for has nothing to do with the sample size.