7
$\begingroup$

In RCTs, randomisation balances unmeasured confounders and, I'm told, ATE and ATT would be the same. In observational studies, this is not possible and Propensity Scores are used in various ways to estimate ATT and/or ATE. The analyses that I've performed and examples that I've seen (eg this helpful text) shows different ATT and ATE (albeit slightly).

Please can anyone help me understand why they are different and, more importantly, what the differences mean (eg. if ATE>ATT or ATT>ATE), if anything?

$\endgroup$
19
$\begingroup$

The Average Treatment Effect (ATE) and the Average Treatment Effect on Treated (ATT) are commonly defined across the different groups of individuals. In addition, ATE and ATT are often different because they might measure outcomes ($Y$) that are not affected from the treatment $D$ in the same manner.

First, some additional notation:

  • $Y^0$: population-level random variable for outcome $Y$ in control state.
  • $Y^1$: population-level random variable for outcome $Y$ in treatment state.
  • $\delta$: individual-level causal effect of the treatment.
  • $\pi$: proportion of population that takes treatment.

Given the above, the ATT is defined as: $\mathrm{E}[\delta|D=1]$ ie. what is the expected causal effect of the treatment for individuals in the treatment group. This can be decomposed more meaningfully as: \begin{align} \mathrm{E}[\delta|D=1] = & \mathrm{E}[Y^1 - Y^0|D=1] \\ & \mathrm{E}[Y^1|D=1] - \mathrm{E}[Y^0|D=1] \end{align}

(Notice that $\mathrm{E}[Y^0|D=1]$ is unobserved so it refers to a counterfactual variable which is not realised in our observed sample.) Similarly the ATE is defined as: $\mathrm{E}[\delta]$, ie. what is the expected causal effect of the treatment across all individuals in the population. Again we can decompose this more meaningfully as: \begin{align} \mathrm{E}[\delta] =& \{ \pi \mathrm{E}[Y^1|D=1] + (1-\pi) \mathrm{E}[Y^1|D=0] \} \\ -& \{ \pi \mathrm{E}[Y^0|D=1] + (1-\pi) \mathrm{E}[Y^0|D=0] \} \end{align}

As you see the ATT and the more general ATE are referring by definition to different portions of the population of interest. More importantly, in the ideal scenario of a randomised control trial (RCT) ATE equals ATT because we assume that:

  • $\mathrm{E}[Y^0|D=1] = \mathrm{E}[Y^0|D=0]$ and
  • $\mathrm{E}[Y^1|D=1] = \mathrm{E}[Y^1|D=0]$,

ie. we have believe respectively that:

  • the baseline of the treatment group equals the baseline of the control group (layman terms: people in the treatment group would do as bad as the control group if they were not treated) and
  • the treatment effect on the treated group equals the treatment effect on the control group (layman terms: people in the control group would do as good as the treatment group if they were treated).

These are very strong assumptions which are commonly violated in observational studies and therefore the ATT and the ATE are not expected to be equal. (Notice that if only the baselines are equal, you can still get an ATT through simple differences: $\mathrm{E}[Y^1|D=1] - \mathrm{E}[Y^0|D=0]$.)

Especially in the cases where the individuals self-select to enter the treatment group or not (eg. an e-shop providing cash bonus where a customer can redeem a bonus coupon for $X$ amount given she shops items worth at least $Y$ amount) the baselines as well as the treatment effects can be different (eg. repeat buyers are more likely to redeem such a bonus, low-value customers might find the threshold $Y$ unrealistically high or high-value customers might be indifferent to the bonus amount $X$ - this also relates to SUTVA). In scenarios like this even talking about ATE is probably ill-defined (eg. it is unrealistic to expect that all the customers of an e-shop will ever shop items worth $Y$).

ATT being unequal to ATE is not unexpected. If ATT is smaller or greater than ATE is application specific. The inequality of the two suggests that the treatment assignment mechanism was potentially not random. In general, in an observational study because the above-mentioned assumptions do not generally hold, we either partition our sample accordingly or we control for difference through "regression-like" techniques.

For a more detailed but easy to follow exposition of the matter I recommend looking into Morgan & Winship's Counterfactuals and Causal Inference.

$\endgroup$
  • $\begingroup$ Thank you very much for this incredibly detailed and helpful answer. I'm not a statistician and struggle at times with formulae, but this is very clear. Will Morga & Winship's book be digestible by a layman, or can you suggest a "dummy's guide" to causal inference? Thanks again $\endgroup$ – bobmcpop Oct 25 '17 at 2:30
  • 1
    $\begingroup$ I am glad I could help. I have educated myself mostly from papers so I have a limited view of what causal inference books are out there. That said, I have found M&W's book to be clear and easy to comprehend; I think an inclined layman will have little problems to follow through. The book is part of the "Analytical Methods for Social Research" series from Cambridge Univ. Press so it uses mostly Sociology-based examples. @DimitriyV.Masterov might have a more educated suggestion. $\endgroup$ – usεr11852 says Reinstate Monic Oct 25 '17 at 23:42
  • $\begingroup$ Thanks, I'll get myself a copy. "The inequality of the two suggests that the treatment assignment mechanism was potentially not random." I assume in a hypothetical situation where literally every baseline confounder was measured in an observational study, and there was a perfect match for each PS, we would get very close to those assumptions. Therefore would the extent to which ATT/ATE are discordant provide any meaningful information about how poorly the PS balanced for unmeasured confounders? $\endgroup$ – bobmcpop Nov 7 '17 at 14:24
  • 1
    $\begingroup$ In a hypothetical situation, yes. I think it would be meaningful in the context of a simulation study. That said, actual using it to quantify "poorness/goodness" of balance achieved by PS is probably a methodical exercise on its own right. (Happy reading!) $\endgroup$ – usεr11852 says Reinstate Monic Nov 7 '17 at 20:56
  • 1
    $\begingroup$ @bobmcpop I'm a statistician, and I never have understood why someone would want to measure the ATT instead of the ATE. It's important to have a control group to account for regression to the mean effects & other factors - you're missing this when you use the ATT. $\endgroup$ – RobertF Oct 18 '18 at 20:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.