# Which one of these methods ATE, ATT, ATO (overlap) should be used to evaluate a strategically applied treatment?

Lets assume I have data from a medical trial in which a new medicine was given to patients who, based on certain characteristics, doctors believed that they would benefit from this new treatment. I want to evaluate

1. Is this new medicine effective?
2. Are doctors effectively choosing the population to provide the new medicine? In other words, how well does the medicine help the people they treated.

I have estimate propensity scores for the population but not sure which methods should be used or if I should be doing matching or weighting.

I am guessing that #1 can be answered by ATE or ATO and #2 can be answered by ATT and maybe ATO?

I'm writing a paper on this, and I'll update with a link when it's posted. (Edit: Here is the arxiv version.) I also discuss the motivation for different estimands in this answer. I'll briefly summarize when each estimand would be substantively useful.

• ATT: the effect of withholding treatment, preventing an exposure, or continuing to implement an experimental program
• ATC: the effect of expanding treatment to those not receiving it or exposing a pollutant to those not exposed
• ATE: the effect of unilaterally implementing one policy vs. unilaterally implementing another; useful when you have no information for making a more nuanced decision
• ATO: the effect of a treatment for those under equipoise, i.e., total uncertainty about which treatment is more effective.

The question of "Is this new medicine effective?" is too broad to be of much use. It might be effective for some and not others. If you want to ask whether it is effective for anyone, you are engaging in "treatment effect discovery", discussed in Mao et al (2018), for which the ATO is best suited because its estimates are precise and less susceptible to bias. This is typically the question small randomized control trials seek to answer. If you want to ask, "Should this medicine be unilaterally recommended over a competing medicine?" then you want the ATE (e.g., comparing name-brand to generic). If you want to ask "Is the medicine effective for those who are already being prescribed it?" then you want the ATT. If you want to ask "Would the medicine be effective for those who not currently being prescribed it?" then you want the ATC. The ATO, ATT, and ATC assume there is already a stable mechanism for treatment prescription in the population.

The second question, "How well does the medicine help the people they treated?" is definitely an ATT question. This would be useful for evaluating whether the medicine should be withheld from those receiving it (i.e., because doctors are not prescribing it well). The more nuanced question of "How can doctors optimally prescribe treatment to optimize results?" is a different question that is not answerable in the framework of the four main estimands. Literature on precision medicine and individual treatment rules may be helpful there.

Different methods target different estimands. I'll summarize these briefly:

• ATT and ATC: pair matching w/o a caliper, full matching, subclassification, weighting by the odds
• ATE: full matching, subclassification, inverse probability weighting
• ATO: pair matching w/ a caliper, cardinality matching, coarsened exact matching, overlap weighting, trimming

If you are using MatchIt or WeightIt in R you can select the estimand you want for each method if available. All weighting methods can be adapted to the ATT, ATC, or ATE, but most matching methods are restricted in the estimand they target. When using full matching or subclassification, make sure you select the correct estimand and use the resulting weights correctly as described in the MatchIt vignette on estimating effects (i.e., rather than estimating separate effects within each subclass).

• Thx this is very useful. If you were creating propensity scores for the population, would you create different models for (or at least control for) different hospitals since their prescription strategies could be very different ? – user3408780 Jun 7 at 2:25
• Yes! There is a literature on multilevel propensity scores that considers this, usually in the context of students within schools. – Noah Jun 7 at 4:56
• Do you have a proof of how to get the propensity weights for ATT? Confused about how these are derived. – user3408780 Jul 8 at 1:41
• I do not. There are some less than satisfying proofs, see Hirano et al. (2003) for example. – Noah Jul 8 at 4:28