Potential Outcome (PO) vs Directed Acyclic Graph (DAG)

Question

Recently I encountered this article of Prof. Imbens (https://arxiv.org/pdf/1907.07271.pdf). It address the capabilities of DAGs for causal inference in comparison with PO. I’m interested in opinions about the fairness of his position.

I give my main points in my answer. I can try to defend what I write but I can go wrong, I stay here for understand more from other opinions.

Answer 1

Initially let me said that this article seems me bad focused. The right comparison should be between SCM vs PO not DAG vs PO. DAGs are part of SCM: SCM = causal graph + SEM seems me a fair simplification. Focusing on DAGs only is a unfair comparison. Indeed even in the past the comparison was: SCM vs PO (https://ftp.cs.ucla.edu/pub/stat_ser/r391-reprint.pdf). Moreover Pearl already said (see article in previous link – par 4) that this debate is rhetorical because SCM and PO are logically equivalent, even if SCMs can be clearer and easier to communicate.

However I have some precise comment about Imbens article.

From paragraph 4

[DAGs are good for exposition but] I am less convinced that the formal identification results are easier to derive in the DAG framework for the type of problems commonly studied in economics. I see three reasons for that.

This is a key point. This position of Imbens seems me strange because causality is bad treated in econometrics literature, even some PO Authors write about it. Indeed in most econometrics books clear identification rules for causal inference are absent, even if such identification seems to be one goal (see here: How would econometricians answer the objections and recommendations raised by Chen and Pearl (2013)?). Indeed It is hard to find even only one econometrics book where such identification rules are exhaustively defined. However the reason for that position are:

First, the DAGs have diffculty coding shape restrictions such as monotonicity …

Indeed DAGs was ideated precisely for do not commit the researcher about some functional form restrictions.

Second, the advantages of the formal methods for deriving identification results with DAGs are most pronounced in complex models with many variables that are not particularly popular in empirical economics

I disagree, It seems me that models with many variables are at the core of economics; Indeed SEMs are focuse on (see here: https://onlinelibrary.wiley.com/doi/10.1002/9781119154136.ch2). If relevant recent papers about SEMs are hard to find, and I have doubts about, it is a researchers fault only. Indeed is precisely PO, at least as usually presented, that seems unequipped for such situations.

Third, the PO approach has connected well with estimation and inference issues. Although the PO approach has largely focused on the problem of estimating average effects of binary treatments, it has been able to make much progress there, not just in terms of identification, but also on problems regarding study design, estimation and inference. On this issue that the PO or Rubin-Causal-Model framework is much more closely tied to statistics and practical issues in inference

The point here is that the progress evocated can achieve results already given in SCM language, no more. Indeed this can be seen as weak point of PO, at least as usually presented and used. Moreover the closeness of PO with estimation seems me not a strogness but another weakness of PO in comparison of SCM. Indeed this fact have much o do with lack of language; probably It is at the origin of ambiguities and contradiction frequently emerged in literature (see here: How would econometricians answer the objections and recommendations raised by Chen and Pearl (2013)?).

The causes are conceptually tied to, at the very least hypothetical, experiments … I find the position [of Pearl] that the manipulation is irrelevant unsatisfactory, and find the insistence in the PO approach on a theoretical or practical manipulation helpful. I am not sure what is meant by $do(obesity = x)$ if the effect of changing obesity depends on the mechanism (say, diet, surgery, or exercise), and the mechanism is not specified in the operator. It is also not obvious to me why we would care about the value of $do(obesity = x)$ if the effect is not tied to an intervention we can envision. The insistence on manipulability in the PO framework resonates well in economics where policy relevance is a key goal (e.g., [Manski, 2013b]). We are interested in policies that change the weight for currently obese people (e.g., encouraging exercise, dietary changes, or surgery), or that discourage currently non-obese people who are at risk of becoming obese from doing so (exercise, dietary changes, or other life-style changes). What is relevant for policy makers is the causal effect of such policies, not the effect of a virtual intervention [$do(obesity = x)$] that makes currently obese people suddenly like non-obese people. … Without a specific manipulation in mind, it is also diffcult to assess a particular identification strategy.

I think that there was a general misunderstandings about the necessity of “manipulation”. I think that Pearl means two things about it. First, the concept of causality do not need experiments, causality exist regardless them. Pearl is right, it seems me obvious. Second, perform actually an experiment is not always needed for causal inference, observational data plus causal assumptions is enough. Indeed even economists have similar idea; Indeed PO authors deal with “hypothetical experiments” (as if), then “hypothetical manipulation”. Indeed most economics study are observational, if this framework is not good for causal inference causality have to be deleted from economic reasoning, an unacceptable restriction I suppose. So all Imbens argument should be revised in the much more slippery ground of what is the difference between “hypothetical experiments” (good) and “virtual intervention” (bad). This distinction can seems semantical. It seems me that what Imbens what mean is that the intervention should be realistic and not just an abstraction while for Pearl a pure abstraction is enough. In my opinion the position of Pearl is the better, a theoretical tool should not have such restriction. It seems me that we can "envision" the intervention even if it is unrealistic. Moreover some intervention that seems unrealistic today can became realistic tomorrow. Said that, scritiny the practical relevance of some causal inference is entirely another question.

Moreover it is not the do-operator the tool that have to specify the “mechanism” of the virtual intervention, or at least not in isolation. The entire SCM specification stands for that and do-operator is senseless without. Moreover the same entire set of assumptions (SCM) is what we have to use for identification and not the do-operator in isolation.

DAGs are by their very definition not cyclical, and as such do not naturally capture assumptions about equilibrium behavior, although there is recent work going in that direction ([Forre and Mooij, 2019]). Equilbrium assumptions are of course central to economics.

Indeed DAGs represent only convenient cases, but SCMs is a broader class. If we have to deal with simultaneity we have to use cyclic graphs and/or simultaneous SEM. This problem is not new in Pearl (and his colleagues) literature, but it his not considered in The Book of Why (divulgative) because too advanced.

However, the implicit assumption that there is no effect of attitudes towards safety and health-related measures on smoking seems very implausible. In all the discussions of this example, Pearl never discusses whether this assumption is plausible. In order to use these methods, one needs to carefully consider every absent link, and in a setting with as many variables as there are in Figure 13(b) that is a daunting task. … In the end a DAG like Figure 8(a), possibly augmented with some arrows between the pre-treatment variables would be just as plausible as any such alternative.

Indeed missed links are strong causal assumptions, researcher have to think carefully about those. DAGs, and SCMs in general, are tools for codify clearly causal assumptions not for avoid them. The advantage of SCM vs PO is that SCM have more and clearer tools for this scope. Indeed the problems underscored by Imbens are more against PO that SCM. Indeed are precisely the obscurities sometimes present in some works based on PO that make assumptions unclear. Causal graphs usually help much about there. In any case difficulties about credibility of some assumptions cannot be resolved from any codification rules, what SCM or PO can give us.

From conclusion

TBOW and the DAG approach fully deserve the attention of all researchers and users of causal inference as one of its leading methodologies. Is it more than that? Should it be the framework of choice for all causal questions, everywhere, or at least in the social sciences, as TBOW argues? In my view no, it should not. The questions it currently answers well are not the ones that are the most pressing ones in practice. Conversely, for the most common and important questions the PO framework is in my view an attractive one in social sciences, and one that resonates well with economic theory by effectively incorporating restrictions beyond conditional independencies.

As said before PO vs DAG is not the right comparison, it should be PO vs SCM; this misunderstanding bring out of road all the discussion. Moreover to refers so much on TBOW is not fair in my opinion. TBOW is only a divulgative book, critics should be study carefully Pearl (2009) if they want to rebut Pearl Theory (SCM). I think that It is very good one and that econometricians, before or after, must surrender and use SCMs.

Moreover, for the problems where DAGs could contribute substantially, the most important issue holding back the DAGs is the lack of convincing empirical applications. History suggests that those are what is driving the adoption of new methodologies in economics and other social sciences, not the mathematical elegance or rhetoric. It is studies such as [LaLonde, 1986], [Card, 1990], [Angrist, 1990], [Angrist and Krueger, 1991], and [Ashenfelter and Krueger, 1994] that spurred the credibility revolution and the adoption of the PO framework, not the theoretical advances – they come later. There have not been similar applications of the DAG framework, and more papers discussing toy models will not be suffcient to convince economists to use this framework.

The “history argument” seems me inconsistent. Indeed, even because SCMs are younger than PO, we can simply said that SCMs was not used by economists for ignorance. Moreover Pearl said repeatedly that SCM and PO can be seen as two different language with the same capabilities; they are logically equivalent but SCMs are more clear. Therefore all relevant results obtained with PO can be obtained with SCM too, maybe in clearer way.