Relevance of baseline covariates in the reference group when estimating ATT

Question

Let's say we have a single treated group A, and then two reference groups B and C. The two reference groups differ systematically in their distribution of one or more baseline covariates (which are correlated with the outcome). Because the ATT is the treatment effect with respect to the distribution of baseline covariates in the treated group only, will the ATT comparing A versus B be the same as the ATT comparing A versus C? Alternatively, does the distribution of baseline covariates in the reference group matter when estimating the ATT in this example?

Edit: This is a non-randomized experiment.

Answer 1

The distribution of the covariates in the reference groups does not have anything to do with the definition of the ATTs for group A, which means you would expect the ATT for A vs. B to be the same as A vs. C if B and C are both the same placebo. This is because the ATTs for group A are $$ E[Y^A|T=A] - E[Y^B|T=A] $$ and $$ E[Y^A|T=A] - E[Y^C|T=A] $$ That is, in an ideal world, the way to compute the ATTs for treatment A would be to take the people who received treatment A, go back in time, and give them treatment B (or C) instead, and examine the outcomes under the truth vs. under the alternate timeline. Nowhere do the people who received treatments B or C in the original world come into this definition.

In practice, we need to use people in treatments B and C to imitate would would have happened to units who received treatment A had they instead received treatments B or C. One way to do this is to manipulate the samples for groups B and C to resemble group A, e.g., using matching or weighting.

The distribution of the covariates in groups B and C affects how successful the matching or weighting will be, but it does not come into the definition of the ATT. For example, if group B is very dissimilar from group A, it may be challenging or impossible to estimate the ATT of A vs. B for group A without extrapolation. Again, that has nothing to do with the definition of the ATT, but it has to do with the feasibility of estimating it from your particular sample.