Interpreting the Results of a Placebo Refutation Test in the Context of Double/Debiased ML Model (DoWhy) Say I run a double/debiased ML model which estimates the casual impact of a treatment variable T on Y as being +\$100 with a 95% C.I. of +/- $10. Attempting to refute this estimate, I run the model again on a random placebo variable, P. The estimate of the casual impact of P on Y +\$25.
What do these results say about my model? Is the fact that the causal estimate of the placebo variable is much larger than zero a sign that the model suffers from misspecification and/or heterogeneity bias? My primary motivation for using a double/debiased ML model is to avoid this exact scenario.
What do these results say about my model's estimates? The original estimate was +\$100, but if the results from the placebo refutation test suggest that the model is overall biased +\$25, is it fair to say that a more realistic estimate of the casual impact of treatment variable T on Y is closer to $75?
I am new to both double/debiased ML modeling as well as placebo refutation tests, so any guidance on their interpretation (as well as any recommended reading materials) would be greatly appreciated.
Thanks!
 A: I think you need a better explanation of what you mean by a placebo - is that an x variable that you know is not a cause of y (a negative control exposure)? An example of this might be father’s smoking, when your primary interest is the effect of mother’s smoking on infant birthweight - you think there may be shared unobserved factors that confound the relationships between both father’s/mother’s smoking, but you are ready to believe that the effect of father’s smoking is at least approximately zero. In this case you can substitute father’s smoking for mother’s smoking in your model - any apparent effect of father’s smoking is then an indication that your results are probably biased.
Or do you have an outcome that you know is not caused by your main exposure (a negative control outcome)? An example of this might be the effect of flu vaccination on non- flu mortality- in this case non-flu mortality is a negative control outcome, as there is no credible mechanism for the flu vaccine to cause it, but it is likely to share confounders (such as health seeking behaviour) with your outcome of primary interest - flu related mortality. Again, any indication that flu vaccine is linked to non- flu mortality is evidence that your main analysis is biased.
Either way, there must be plausible shared confounding, and the approach can never rule out non-shared confounders - i.e. there may be confounders of the relationship between mother’s smoking and birthweight that are specific to the mother and unrelated to the father’s smoking. There are methods you can use to try to adjust for confounding using negative controls, but none will get around that issue. In short- this means that you probably should not believe that the true effect is +75 in your example. I should also emphasise that the assumptions rely on your substantive knowledge of the topic to rule out alternative explanations and identify useful negative controls. Finally, none of this is specific to a particular method of adjustment.
