0
$\begingroup$

In an ideal balanced randomized experiment where every profile's propensity score is 0.5 (under binary treatment), regardless of the value of the profile, the best we can know of the treatment effect(AKA uplift) of the given subject is ATE (Average Treatment Effect) of the entire group: This is my understanding.

If this reasoning is correct, what is the point of using decision tree to estimate treatment effect of the subject with profile X because the ATE at the leaf node will be (statistically) same as that of the entire group.

The following is the relevant survey paper, where I could not find answer to my above doubt: http://proceedings.mlr.press/v67/gutierrez17a/gutierrez17a.pdf

$\endgroup$
1
$\begingroup$

It's not generally the case that the conditional average treatment effect (CATE) for a given profile is equal to the average treatment effect in the sample (ATE). In fact, the average of the CATEs in the sample will equal the ATE, but if the CATEs differ from each other across profiles, then the CATE for a given profile won't necessarily be equal to the average of the CATEs. This situation is called "treatment effect heterogeneity" because the treatment effects (CATEs) are heterogeneous.

This has nothing to do with whether a randomized experiment was performed or not. In a randomized experiment, the ATE is given for free by computing the difference in observed outcome means, and in an observational study with confounding, we have to do analytic work to get the ATE (which is what the article describes). To get the CATE for a given profile, however, we have to do analytic work regardless of whether we have a randomized experiment or an observational study. In fact, this analytic work is almost the same in both cases, except for the situation mentioned in the paper you linked, in which an alternative and simpler method arise for estimating CATEs in a randomized experiment that would not be available in an observational study. But all methods for estimating CATEs in observational studies apply to randomized experiments as well.

So, to answer your specific question, it's not the case that CATEs in leaf nodes will equal the ATE in the sample. On average they will, but only because by definition the ATE is the average of the CATEs. For a given profile, you wouldn't expect the CATE to be equal to the ATE unless the treatment effect is exactly equal for all profiles.

For example, let's say an intervention performed better for younger than older customers (e.g., because it relies on social media). In a randomized experiment, there would be the same number of older and younger customers in each condition of the intervention (i.e., with the intervention present vs. absent). You could compute the ATE by taking the difference in sample means for the two conditions, but to estimate the CATEs for young and old customers, additional modeling would have to take place (which could be as simple as stratifying by age). You wouldn't expect the CATEs for these age profiles to be equal to the overall ATE based on the first premise of this example. The techniques for estimating CATEs in the article rely on the same reasoning except that the groups are more granular and harder to observe directly, so modeling (including machine learning) has to be done to get good CATE estimates.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.