# How to compare treatments in multitreatment setting of observational treatment study

I am researching uplift models to measure effect of treatment. Particularly when there are multiple treatments and I want to compare/order treatments based on their causal effect on average/individual. In literature (e.g. here study) authors often compare each treatment to control group and estimate effect, then order based on this effect. In case of observational data one must correct heterogeneous treatment groups with e.g. propensity score matching - where individuals from control group are matched to treated individuals based on similarity to ensure homogeneous subjects in each group.

Now to my question. As is the case with observational data, the subjects in each group may be heterogeneous. For example, younger people are assigned more frequently to treatment A, while older people to treatment B, control group is randomly assigned.

Now if I do matching between treatment A and control, I will sample mostly young people from control. Same with old people with treatment B and control. Now let say I am interested in effect on marketing ad A and ad B and no ad - control. Generally (assumption for this example) younger people are more likely to respond to online ad, while older are less likely. If no ad is shown the effect of age is negligible. If either add is shown, younger people will respond more. Thus effect of ad A will be much higher than effect of ad B, since most of young people are assigned to ad A. This is not causal effect of the treatment as it is biased by the age of the groups. It only compares treatment to control. It cannot be used to compare treatment A to treatment B.

Am I overlooking something? What is the correct way to compare treatment A to treatment B?

Thank you

• You don't need to use propensity score matching. In fact, for this type of problem, it is a bad method to use. Weighting or regression would be far more effective.
– Noah
Apr 14, 2020 at 17:56
• Yes, after submiting the question, I realized inverse propensity weighting would not have this kind of problem. Would you please explain what do you mean by regresion method that would be effective @Noah ? Apr 14, 2020 at 18:34
• Just regress the outcome on the covariates and treatment. Of course there are more sophisticated methods of doing this. See this paper for examples.
– Noah
Apr 14, 2020 at 19:10

So what I hear you saying is that, while you're really interested in the effect of the treatment $$T$$ on the outcome $$O,$$ there is another variable age, $$A,$$ that you think is likely confounding the results. You're absolutely right. Here is a causal diagram, as per Judea Pearl's ideas:
There is a backdoor path from $$T$$ to $$O:$$ $$T\leftarrow A\to O.$$ Therefore, you must condition on $$A$$ to obtain the correct causal effect of $$T$$ on $$O.$$ You can use the backdoor adjustment formula: $$P(O=o|\operatorname{do}(T=t))=\sum_aP(O=o|T=t,A=a)\,P(A=a).$$ Here the first expression is saying, "What is the probability that I would get outcome $$o$$ if I set $$T=t?$$ That's what that '$$\operatorname{do}$$' operator expresses. In words, you need to adjust, or condition, on the age to get the correct causal effect of the treatment.
• It doesn't matter if $A$ does not influence $O$ if there is no treatment. It does influence it if there is a treatment. The arrow in these diagrams doesn't mean that the first node always influences the second; it means the first node can influence the second. In this case, because age is a confounding variable, you cannot properly analyze the data in the aggregate. You must segregate out by age. Apr 15, 2020 at 15:17