64
$\begingroup$

In the year 2000, Judea Pearl published Causality. What controversies surround this work? What are its major criticisms?

$\endgroup$
3
  • 11
    $\begingroup$ There's an informative discussion in the archives of Andrew Gelman's blog, including contributions from Pearl and other experts. $\endgroup$ – guest Apr 14 '12 at 4:01
  • 12
    $\begingroup$ Gelman discusses Pearl's Causality, in addition to SL Morgan and C Winship's Counterfactuals and Causal Models and A Sloman's Causal Models in a 2011 review essay in the Am. J. of Sociology. He is generally very supportive of Pearl's contributions, especially Pearl's formalization of causal models in terms of interventions (do-calculus). However, he remains concerned that state-of-the-art causal theory may still invite oversimplified causal models and subsequently false causal inferences from observational data. $\endgroup$ – jthetzel Aug 7 '12 at 15:06
  • 1
    $\begingroup$ @jthetzel: Thanks, that looks like a good answer to me. Would you mind adding it? $\endgroup$ – Neil G Aug 8 '12 at 19:17
43
$\begingroup$

Some authors dislike Pearl's focus on the directed acyclic graph (DAG) as the way in which to view causality. Pearl essentially argues that any causal system can be considered as a non-parametric structural equation model (NPSEM), in which the value of each node is taken as a function of its parents and some individual error term; the error terms between different nodes may in general be correlated, to represent common causes.

Cartwright's book Hunting Causes and Using Them, for example, gives an example involving a car engine, which she claims cannot be modelled in the NPSEM framework. Pearl disputes this in his review of Cartwright's book.

Others caution that the use of DAGs can be misleading, in that the arrows lend an apparent authority to a chosen model as having causal implications, when this may not be the case at all. See Dawid's Beware of the DAG. For example, the three DAGs $A \rightarrow B \rightarrow C$, $A \leftarrow B \rightarrow C$ and $A \leftarrow B \leftarrow C$ all induce the same probabilistic model under Pearl's d-separation criterion, which is that A is independent of C given B. They are therefore indistinguishable based upon observational data.

However they have quite different causal interpretations, so if we wish to learn about the causal relationships here we would need more than simply observational data, whether that be the results of interventional experiments, prior information about the system, or something else.

$\endgroup$
3
  • 5
    $\begingroup$ To be fair - far from being unaware of the three DAGs with the same probabilistic model, Pearl has been one of the chief promoters of the distinction between merely statistical-probabilistic-associational models and fully causal models. See for example Section 2 of ftp.cs.ucla.edu/pub/stat_ser/r354-corrected-reprint.pdf $\endgroup$ – Paul Dec 11 '17 at 18:02
  • 1
    $\begingroup$ @Paul yes indeed; I was just reporting other people's misgivings about using DAGs. I have no such misgivings - please edit if you think the reply is unfair! $\endgroup$ – rje42 Dec 13 '17 at 9:16
  • 10
    $\begingroup$ It just sounds like the message was totally lost in translation. Which is not necessarily the fault of your answer, if you're just reporting the criticisms people have made. The whole point of Pearl's work is that different causal models can generate the same probabilistic model and hence the same-looking data. So it's not enough to have a probabilistic model, you have to base your analysis and causal interpretation on the full DAG to get reliable results. If you're just reporting what people say, I don't think your answer needs editing, these comments are sufficient clarification. $\endgroup$ – Paul Dec 13 '17 at 16:58
21
$\begingroup$

I think this framework has a lot of trouble with general equilibrium effects or Stable Unit Treatment Value Assumption violations. In that case, the "untreated" observations no longer provide the desired counterfactual in a meaningful way. Massive job training programs that shift the entire wage distribution are one example. The counterfactual may not even be well-defined in some cases. In Morgan and Winship's Counterfactuals and Causal Models, they give an example of the claim that the 2000 election would have gone in favor of Al Gore if felons and ex-felons had been allowed to vote. They point out that the counterfactual world would have very different candidates and issues, so that you cannot characterize the alternative causal state. The ceteris paribus effect would not be the policy relevant parameter here.

$\endgroup$
3
  • $\begingroup$ It sounds like you're saying that some counterfactuals are not reasonable because it's not reasonable to assume that only one thing changes? In the felon example, the simple fact of felons being able to vote would imply many other differences between that potential world and our actual world, so it's not reasonable to change "just one thing"? $\endgroup$ – Paul Dec 11 '17 at 17:59
  • 3
    $\begingroup$ @Paul Yes, the "all else equal" cannot hold. $\endgroup$ – Dimitriy V. Masterov Dec 11 '17 at 22:45
  • 2
    $\begingroup$ Thanks. I think this is a fairly profound and underappreciated point about counterfactuals. People usually assume they can do whatever they want. But just like the real world, I guess the space of valid counterfactuals can have "multicollinearity". $\endgroup$ – Paul Dec 11 '17 at 23:03
4
$\begingroup$

The most important criticism of Pearl's system is, from my perspective, that it has not yielded any practical, empirical advances anywhere it has been used. Given how long it has been around, there's no reason to think it will ever be a practical tool. This indicates that it can be used for some theoretical and perhaps didactic purposes, but a practical researcher will gain little from studying it.

$\endgroup$
5
  • 2
    $\begingroup$ A warm welcome to this site, but your answer is totally ridiculous. $\endgroup$ – Neil G May 22 '19 at 8:00
  • 4
    $\begingroup$ Why is it ridiculous? If Pearl promoted his system simply as some kind of conceptual, philosophical tool for understanding what causality is, I wouldn't have a problem with it. But he constantly talks it up as a "revolutionary" practical tool for researchers to use, which is simply bullshit. For example, in his latest book Pearl says that he "would not be surprised" if the front-door method "eventually becomes a serious competitor to randomized controlled trials", which is a strong claim given that there's not a single example of the method being used to solve any real problem, ever. $\endgroup$ – Matt May 22 '19 at 8:27
  • 3
    $\begingroup$ It's ridiculous because his work has been cited tens of thousands of times. The front-door method was famously used to support the link between smoking and cancer in defiance of Ronald Fisher's testimony! $\endgroup$ – Neil G May 22 '19 at 8:47
  • 15
    $\begingroup$ What does Pearl's citation count have to do with anything? My criticism is that the practical benefits that he has promised for decades have not materialized. Pearl came up with the front-door criterion decades after Fisher had died and the cancer-and-smoking controversy had settled down. How could the criterion have been used against Fisher? $\endgroup$ – Matt May 22 '19 at 9:09
  • 1
    $\begingroup$ @Matt; I disagree with your point, I explain why in my answer. $\endgroup$ – markowitz Dec 10 '20 at 9:16
4
$\begingroup$

Disclaimer 1: I am a fan of Pearl's framework, and had the privilege of being taught counterfactual formal causal inference by two of his early exponents (Hernán and Robins).

Disclaimer 2: Levins was my mentor when I was a doctoral student, and I have published using his methods.

The counterfactual theory of causality, and the counterfactual formal causal inference/reasoning built atop of it, are profoundly useful for reasoning through both the strengths and weaknesses of causal inference based on specific combinations of study design and analyses. However, to my mind, the counterfactual theory is a theory of terminal causal narratives: $A$, and $L$, and $U$, (and maybe $V$ or $E$) happened, and then the caused $\pmb{Y}$ to happen (or not). However, the counterfactual theory of causality does not appear to describe or infer the behavior of complex causal systems (i.e. networks in which every variable is either directly or indirectly the cause of every other variable at some future time), and is thus not a theory of cyclic causal narratives.

I would raise as a counterexample of a formal causal reasoning system Levins' qualitative loop analysis, which, like Pearl's work with DAGs also hearkens back to Wright's path analysis, but employs signed digraphs in a different causal formalism (in fact, one obvious distinction is that qualitative loop analysis employs signed digraphs which are cyclic, not acyclic), to describe the behavior of such causal systems under different kinds of perturbation.

Simple two-variable signed digraph, possibly representing a trophic relationship

The questions posed and answered by Levins' method (and subsequent elaborations on it) include:

  • How does the level of each variable in a complex system respond to press perturbation at one or more variables in the system?
  • How does the life expectancy/turnover of each variable in a complex system respond to press perturbation at one or more variables in the system?
  • Does variance induced by system perturbation (at specific variables) tend to diffuse across the system, or sink into very few variables in the system?
  • Where do (Lyapunov) stability and instability emerge in the system?
  • Where does system behavior depend on either ontological or epistemic uncertainty regarding the existence or magnitude of specific direct causal relationships comprising the system?
  • What are the signs of expected bivariate correlations (or correspondences) between any variable pairs given a press perturbation at one or more variables in the system?

(Because most of Levins' loop analysis is a purely deductive method—although, see Dambacher's extension—only the bolded question is directly statistical.)

These questions are different questions than the ones posed and answered in the counterfactual formal causal inference championed by Pearl. I have even had difficulty finding examples of counterfactual formal causal inference applied in the context of stochastic processes and autoregressive models (e.g., dynamic models including $Y_{t}$ as a function of $Y_{<t}$), although this may be more due to my lack of familiarity with the intersections of Bayesian probabilistic causal graphs and Pearl's work, than due to a specific deficiency in the latter.

Aside: Sugihara's empirical dynamic modeling (see tutorial by Chaing, et al.) elaboration on state space reconstructions, likewise provides an alternative perspective to counterfactual formal causal reasoning, also from the world of complex causal systems.

Video of Sugihara and friends blowing your mind and mine by recovering topology of 3D complex system from 1D time series.

References

Chang, C.-W., Ushio, M., & Hseih, C. (2017). Empirical Dynamic Modeling for Beginners. Ecological Research, 32(6), 785–796.

Dambacher, J. M., Li, H. W., & Rossignol, P. A. (2003). Qualitative predictions in model ecosystems. Ecological Modelling, 161, 79 /93.

Dambacher, J. M., Levins, R., & Rossignol, P. A. (2005). Life expectancy change in perturbed communities: Derivation and qualitative analysis. Mathematical Biosciences, 197, 1–14.

Levins, R. (1974). The Qualitative Analysis of Partially Specified Systems. Annals of the New York Academy of Sciences, 231, 123–138.

Puccia, C. J., & Levins, R. (1986). Qualitative Modeling of Complex Systems: An Introduction to Loop Analysis and Time Averaging. Harvard University Press.

Sugihara, G., May, R., Ye, H., Hsieh, C., Deyle, E., Fogarty, M., & Munch, S. (2012). Detecting Causality in Complex Ecosystems. Science, 338, 496–500.

Wright, S. (1934). The Method of Path Coefficients. The Annals of Mathematical Statistics, 5(3), 161–215.

$\endgroup$
1
  • 1
    $\begingroup$ @DimitriyV.Masterov I would so love your perspectives here! $\endgroup$ – Alexis Mar 2 at 18:22
3
$\begingroup$

Reading answers and comments I feel the opportunity to add something.

The accepted answer, by rje42, is focused on DAG’s and non parametric systems; strongly related concepts. Now, capabilities and limitations of these tools can argued, however we have to say that linear SEMs is part of the Theory presented in Pearl manual (2000 or 2009). Pearl underscore limitations of linear systems but them are part of the Theory presented.

The comment of Paul seems me crucial: “The whole point of Pearl's work is that different causal models can generate the same probabilistic model and hence the same-looking data. So it's not enough to have a probabilistic model, you have to base your analysis and causal interpretation on the full DAG to get reliable results.” Let me says that the last phrase can be rewritten as: so it's not enough to have a probabilistic model, you need structural causal equation/model (SCM). Pearl said us to keep in mind a demarcation line between statistical and structural/causal concepts. The former alone can be never enough for proper causal inference, we need the latter too; here stay the root of most problems. In my opinion this clear distinction and his defense, represent the most important merit of Pearl.
Moreover Pearl suggests some tools as: DAG, d-separation, do-operator, backdoor and front door criterion among others. All of them are important, and express his Theory, but all come from the demarcation line remembered above, and all help us to work according with it. Put it differently, is not so tremendously relevant to argue pro or cons one specific tool while it is about the necessity of the demarcation line. If the demarcation line disappear all Pearl Theory go down or, at best, add just a bit of language on what we already have. However it seems me an unsustainable position. If some authors, today yet, seriously argue so, please give me some reference about.

I'm not yet expert enough for challenge the capability of all these tools but them seems me clear and, until now, it seems me that them work. I come from econometric side and, about therein tools, I think the opposite. I can said that econometrics is: very widespread, very useful, very practical, very empirical, very challenging, very considered matters; and It have one of his most interesting topic in causality. In econometrics some causal issues can be fruitfully addressed with RCT tools. However, unfortunately, we can show that econometrics literature, too often, addressed causal problems not properly. Shortly, this happened due to theoretical flaws. The dimensions of this problem emerge in their endemicity in:

Regression and Causation: A Critical Examination of Six Econometrics Textbooks - Chen and Pearl (2013) and

Trygve Haavelmo and the Emergence of causal calculus - Pearl; Econometric Theory (2015)

In these related discussions some point are addressed:

Under which assumptions a regression can be interpreted causally?

Difference Between Simultaneous Equation Model and Structural Equation Model

I don’t know if "equilibrium problems" invoked by Dimitriy V. Masterov cannot be addressed properly with Pearl SCMs, but from here:

Eight Myths About Causality and Structural Equation Models - Handbook of Causal Analysis for Social Research, Springer (2013)

emerge that some frequently invoked limitations are false.

Finally, the argument of Matt seems me not relevant, but not for "citations evidence" as argued by Neil G. In two words the Matt point is

“Pearl's theory can be good for itself but not for practice purpose”.

It seems me a logically wrong argument, definitely. Matter of fact is that Pearl presented a Theory. So, suffice to say the motto “nothing can be more practical and useful than a good theory”. Said that it is obvious that the examples in the book are simple as possible, good didactic practice demand it. Making the things more complicated is always possible and not difficult, at the opposite proper simplifications are usually hard to make. The possibility to face simple problems or rend them more simple seems me a strong evidence in favor of Pearl's Theory.

Said that, the fact that no real issues are solved by Pearl's Theory (if it is true) is neither his, or from his Theory, responsibility. Himself complains that Professors and researchers haven't spent time enough on his (and causal inference in general) theory and tools. This fact could be justified only in face of clear theoretical flaw of Pearl's Theory and clear superiority of another one. Curious to see that probably the opposite is true; note that Pearl argued that RCT boil down in SCM. The problem that Matt underscore come from Professors and researchers responsibility.

I think that in the future Pearl's Theory will become common in econometrics too.

$\endgroup$
12
  • 2
    $\begingroup$ ”Matter of fact is that Pearl presented a Theory." What theorems did Pearl present? Is a 'do-operator' a theory? Does this theory make any change to the results when a researcher tests a physical theory at CERN or when a pharmacy company analyses the effectivity of a vaccine? Or does it just allow them to draw pretty diagrams like managers do? $\endgroup$ – Sextus Empiricus Dec 10 '20 at 8:39
  • 2
    $\begingroup$ All things presented in Pearl 2009 represent a Theory, do-operator is relevant therein. This book is plenty of theorems, some referred on Pearl’s works. Graphs is only a language, concepts stay behind them. I’am not expert enough in Physics or pharmacy literature and/or practice for help you to find and discuss example about. $\endgroup$ – markowitz Dec 10 '20 at 9:15
  • 2
    $\begingroup$ But what sort of theory is it? A reference to an entire book does not help much in conveying what this theory is and is an alarm bell that there is no theory after all, and instead just a lot of mumbo jumbo. Is there any theory from Pearl that can be described by a few sentences or by a single formula? And is the theory a scientific theory, which means that it can be tested experimentally or proven mathematically? Or is it just a proposal for a different language, a set of definitions, expressions and methods? Or is it more like philosophy? $\endgroup$ – Sextus Empiricus Dec 10 '20 at 9:48
  • 1
    $\begingroup$ I do not stay here for defence Pearl’s works, he do not need my help. Pearl is an eminent scientist, and his manual was considered one of the most important from some decades to now. You said “A reference to an entire book does not help much in conveying what this theory is and is an alarm bell that there is no theory after all, and instead just a lot of mumbo jumbo”. You are wrong. Books like Pearl (2009) has been written precisely with the scope to present a Theory, but I cannot summarize it for you here. $\endgroup$ – markowitz Dec 10 '20 at 10:37
  • 2
    $\begingroup$ However one “alarm bell” actually is here, it is your tone. Worse, in this discussion stats.stackexchange.com/questions/499455/… you confess: “I haven't read any of Pearl's books”, but anyway you feel the necessity to intervene here and speak against his main contribution. Forgive me for my frankness but your words reveal unjustified dislike towards Pearl and/or some ignorance. $\endgroup$ – markowitz Dec 10 '20 at 10:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.