I am researching uplift models to measure effect of treatment. Particularly when there are multiple treatments and I want to compare/order treatments based on their causal effect on average/individual. In literature (e.g. here study) authors often compare each treatment to control group and estimate effect, then order based on this effect. In case of observational data one must correct heterogeneous treatment groups with e.g. propensity score matching - where individuals from control group are matched to treated individuals based on similarity to ensure homogeneous subjects in each group.

Now to my question. As is the case with observational data, the subjects in each group may be heterogeneous. For example, younger people are assigned more frequently to treatment A, while older people to treatment B, control group is randomly assigned. schema of example

Now if I do matching between treatment A and control, I will sample mostly young people from control. Same with old people with treatment B and control. Now let say I am interested in effect on marketing ad A and ad B and no ad - control. Generally (assumption for this example) younger people are more likely to respond to online ad, while older are less likely. If no ad is shown the effect of age is negligible. If either add is shown, younger people will respond more. Thus effect of ad A will be much higher than effect of ad B, since most of young people are assigned to ad A. This is not causal effect of the treatment as it is biased by the age of the groups. It only compares treatment to control. It cannot be used to compare treatment A to treatment B.

Am I overlooking something? What is the correct way to compare treatment A to treatment B?

Thank you

  • $\begingroup$ You don't need to use propensity score matching. In fact, for this type of problem, it is a bad method to use. Weighting or regression would be far more effective. $\endgroup$
    – Noah
    Commented Apr 14, 2020 at 17:56
  • $\begingroup$ Yes, after submiting the question, I realized inverse propensity weighting would not have this kind of problem. Would you please explain what do you mean by regresion method that would be effective @Noah ? $\endgroup$ Commented Apr 14, 2020 at 18:34
  • $\begingroup$ Just regress the outcome on the covariates and treatment. Of course there are more sophisticated methods of doing this. See this paper for examples. $\endgroup$
    – Noah
    Commented Apr 14, 2020 at 19:10

1 Answer 1


So what I hear you saying is that, while you're really interested in the effect of the treatment $T$ on the outcome $O,$ there is another variable age, $A,$ that you think is likely confounding the results. You're absolutely right. Here is a causal diagram, as per Judea Pearl's ideas:

enter image description here

There is a backdoor path from $T$ to $O:$ $T\leftarrow A\to O.$ Therefore, you must condition on $A$ to obtain the correct causal effect of $T$ on $O.$ You can use the backdoor adjustment formula: $$P(O=o|\operatorname{do}(T=t))=\sum_aP(O=o|T=t,A=a)\,P(A=a).$$ Here the first expression is saying, "What is the probability that I would get outcome $o$ if I set $T=t?$ That's what that '$\operatorname{do}$' operator expresses. In words, you need to adjust, or condition, on the age to get the correct causal effect of the treatment.

For more details, I would strongly encourage you to take a look at The Book of Why, by Pearl and Mackenzie, Causal Inference in Statistics: A Primer, by Pearl, Glymour, and Jewell, and lastly, Causality: Models, Reasoning, and Inference, by Pearl.

  • $\begingroup$ Thank you for answer, but I am not sure.. In my example A does not influence O if there is no treatment. Thus A directly influences the effect. In case of basic confounding as you show on diagram, Propensity matching is often used to deconfound. In my example, matching those treatments with control group will create 2 very different matched datasets with different ratios of young and old people. My question was, how to do matching correctly in this case? E.g. If I force to keep all control data in matching, or ... Anyway I will read into the back door adjustment, thx for suggestio $\endgroup$ Commented Apr 15, 2020 at 7:17
  • $\begingroup$ It doesn't matter if $A$ does not influence $O$ if there is no treatment. It does influence it if there is a treatment. The arrow in these diagrams doesn't mean that the first node always influences the second; it means the first node can influence the second. In this case, because age is a confounding variable, you cannot properly analyze the data in the aggregate. You must segregate out by age. $\endgroup$ Commented Apr 15, 2020 at 15:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.