76
$\begingroup$

In the year 2000, Judea Pearl published Causality. What controversies surround this work? What are its major criticisms?

$\endgroup$
2
  • 12
    $\begingroup$ There's an informative discussion in the archives of Andrew Gelman's blog, including contributions from Pearl and other experts. $\endgroup$
    – guest
    Apr 14, 2012 at 4:01
  • 14
    $\begingroup$ Gelman discusses Pearl's Causality, in addition to SL Morgan and C Winship's Counterfactuals and Causal Models and A Sloman's Causal Models in a 2011 review essay in the Am. J. of Sociology. He is generally very supportive of Pearl's contributions, especially Pearl's formalization of causal models in terms of interventions (do-calculus). However, he remains concerned that state-of-the-art causal theory may still invite oversimplified causal models and subsequently false causal inferences from observational data. $\endgroup$
    – jthetzel
    Aug 7, 2012 at 15:06

6 Answers 6

52
$\begingroup$

Some authors dislike Pearl's focus on the directed acyclic graph (DAG) as the way in which to view causality. Pearl essentially argues that any causal system can be considered as a non-parametric structural equation model (NPSEM), in which the value of each node is taken as a function of its parents and some individual error term; the error terms between different nodes may in general be correlated, to represent common causes.

Cartwright's book Hunting Causes and Using Them, for example, gives an example involving a car engine, which she claims cannot be modelled in the NPSEM framework. Pearl disputes this in his review of Cartwright's book.

Others caution that the use of DAGs can be misleading, in that the arrows lend an apparent authority to a chosen model as having causal implications, when this may not be the case at all. See Dawid's Beware of the DAG. For example, the three DAGs $A \rightarrow B \rightarrow C$, $A \leftarrow B \rightarrow C$ and $A \leftarrow B \leftarrow C$ all induce the same probabilistic model under Pearl's d-separation criterion, which is that A is independent of C given B. They are therefore indistinguishable based upon observational data.

However they have quite different causal interpretations, so if we wish to learn about the causal relationships here we would need more than simply observational data, whether that be the results of interventional experiments, prior information about the system, or something else.

$\endgroup$
5
  • 8
    $\begingroup$ To be fair - far from being unaware of the three DAGs with the same probabilistic model, Pearl has been one of the chief promoters of the distinction between merely statistical-probabilistic-associational models and fully causal models. See for example Section 2 of ftp.cs.ucla.edu/pub/stat_ser/r354-corrected-reprint.pdf $\endgroup$
    – Paul
    Dec 11, 2017 at 18:02
  • 2
    $\begingroup$ @Paul yes indeed; I was just reporting other people's misgivings about using DAGs. I have no such misgivings - please edit if you think the reply is unfair! $\endgroup$
    – rje42
    Dec 13, 2017 at 9:16
  • 16
    $\begingroup$ It just sounds like the message was totally lost in translation. Which is not necessarily the fault of your answer, if you're just reporting the criticisms people have made. The whole point of Pearl's work is that different causal models can generate the same probabilistic model and hence the same-looking data. So it's not enough to have a probabilistic model, you have to base your analysis and causal interpretation on the full DAG to get reliable results. If you're just reporting what people say, I don't think your answer needs editing, these comments are sufficient clarification. $\endgroup$
    – Paul
    Dec 13, 2017 at 16:58
  • $\begingroup$ I think Pearl has all the caveats you've mentioned in his book, actually. In fact, Theorem 1.2.8 in Causality: Models, Reasoning, and Inference, 2nd Ed. is all about observationally equivalent DAGs. A DAG is just part of the model, and is no more authoritative or less authoritative than any other model. $\endgroup$ Oct 29, 2021 at 14:56
  • 1
    $\begingroup$ That third paragraph does not so much seem to be a criticism, as an illustration of the usefulness of DAGs in making the limitations of our ability to infer causes explicit. $\endgroup$
    – Alexis
    May 26, 2022 at 15:07
26
$\begingroup$

I think this framework has a lot of trouble with general equilibrium effects or Stable Unit Treatment Value Assumption violations. In that case, the "untreated" observations no longer provide the desired counterfactual in a meaningful way. Massive job training programs that shift the entire wage distribution are one example. The counterfactual may not even be well-defined in some cases. In Morgan and Winship's Counterfactuals and Causal Inference, they give an example of the claim that the 2000 election would have gone in favor of Al Gore if felons and ex-felons had been allowed to vote. They point out that the counterfactual world would have very different candidates and issues, so that you cannot characterize the alternative causal state. The ceteris paribus effect would not be the policy relevant parameter here.

$\endgroup$
3
  • 4
    $\begingroup$ It sounds like you're saying that some counterfactuals are not reasonable because it's not reasonable to assume that only one thing changes? In the felon example, the simple fact of felons being able to vote would imply many other differences between that potential world and our actual world, so it's not reasonable to change "just one thing"? $\endgroup$
    – Paul
    Dec 11, 2017 at 17:59
  • 5
    $\begingroup$ @Paul Yes, the "all else equal" cannot hold. $\endgroup$
    – dimitriy
    Dec 11, 2017 at 22:45
  • 5
    $\begingroup$ Thanks. I think this is a fairly profound and underappreciated point about counterfactuals. People usually assume they can do whatever they want. But just like the real world, I guess the space of valid counterfactuals can have "multicollinearity". $\endgroup$
    – Paul
    Dec 11, 2017 at 23:03
12
$\begingroup$

Counterfactual formal causal reasoning of the form motivated by Pearl is ill-suited to the analysis of complex dynamic causal systems (i.e. networks in which every variable is either directly or indirectly the cause of every variable at some future time).1

Disclaimer 1: I am a fan of Pearl's framework, and had the privilege of being taught counterfactual formal causal inference by two of his early exponents (Hernán and Robins).

Disclaimer 2: Levins was my mentor when I was a doctoral student, and I have published using his methods.

The counterfactual theory of causality, and the counterfactual formal causal inference/structural causal modeling built atop of it, are profoundly useful for reasoning through both the strengths and weaknesses of causal inference based on specific combinations of study design and analyses. However, to my mind, structural causal models built out of the counterfactual theory is a theory of terminal causal narratives: $A$, and $L$, and $U$, (and maybe $V$ or $E$) happened, and then they caused $\pmb{Y}$ to happen (or not). However, the counterfactual theory of causality does not appear to describe or infer the behavior of complex causal systems, and is thus not a theory of cyclic causal narratives.

I would raise as a counterexample of a formal causal reasoning system Levins' qualitative loop analysis, which, like Pearl's work with DAGs also hearkens back to Wright's path analysis, but employs signed digraphs in a different causal formalism (in fact, one obvious distinction is that qualitative loop analysis employs signed digraphs which are cyclic, not acyclic), to describe the behavior of such causal systems under different kinds of perturbation.

Simple two-variable signed digraph, possibly representing a trophic relationship

The questions posed and answered by Levins' method (and subsequent elaborations on it) include:

  • How does the level of each variable in a complex system respond to press perturbation at one or more variables in the system?
  • How does the life expectancy/turnover of each variable in a complex system respond to press perturbation at one or more variables in the system?
  • Does variance induced by system perturbation (at specific variables) tend to diffuse across the system, or sink into very few variables in the system?
  • Where do (Lyapunov) stability and instability emerge in the system?
  • Where does system behavior depend on either ontological or epistemic uncertainty regarding the existence or magnitude of specific direct causal relationships comprising the system?
  • What are the signs of expected bivariate correlations (or correspondences) between any variable pairs given a press perturbation at one or more variables in the system?

(Because most of Levins' loop analysis is a purely deductive method—although, see Dambacher's extension—only the bolded question is directly statistical.)

These questions are different questions than the ones posed and answered in the counterfactual formal causal inference championed by Pearl. I have even had difficulty finding examples of counterfactual formal causal inference applied in the context of stochastic processes and autoregressive models (e.g., dynamic models including $Y_{t}$ as a function of $Y_{<t}$), although this may be more due to my lack of familiarity with the intersections of Bayesian probabilistic causal graphs and Pearl's work, than due to a specific deficiency in the latter.

Aside: Sugihara's empirical dynamic modeling (see tutorial by Chaing, et al.) elaboration on state space reconstructions, likewise provides an alternative perspective to counterfactual formal causal reasoning, also from the world of complex causal systems.

Video of Sugihara and friends blowing your mind and mine by recovering the topology of 3D complex system from a 1D time series.


1 A point similar to something Spirtes pointed out quite a while ago.


References

Chang, C.-W., Ushio, M., & Hseih, C. (2017). Empirical Dynamic Modeling for Beginners. Ecological Research, 32(6), 785–796.

Dambacher, J. M., Li, H. W., & Rossignol, P. A. (2003). Qualitative predictions in model ecosystems. Ecological Modelling, 161, 79 /93.

Dambacher, J. M., Levins, R., & Rossignol, P. A. (2005). Life expectancy change in perturbed communities: Derivation and qualitative analysis. Mathematical Biosciences, 197, 1–14.

Levins, R. (1974). The Qualitative Analysis of Partially Specified Systems. Annals of the New York Academy of Sciences, 231, 123–138.

Puccia, C. J., & Levins, R. (1986). Qualitative Modeling of Complex Systems: An Introduction to Loop Analysis and Time Averaging. Harvard University Press.

Spirtes, P. (1995). Directed Cyclic Graphical Representations of Feedback. In P. Besnard & S. Hanks (Eds.), Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence. Morgan Kaufmann Publishers, Inc.

Sugihara, G., May, R., Ye, H., Hsieh, C., Deyle, E., Fogarty, M., & Munch, S. (2012). Detecting Causality in Complex Ecosystems. Science, 338, 496–500.

Wright, S. (1934). The Method of Path Coefficients. The Annals of Mathematical Statistics, 5(3), 161–215.

$\endgroup$
11
  • 2
    $\begingroup$ @DimitriyV.Masterov I would so love your perspectives here! $\endgroup$
    – Alexis
    Mar 2, 2021 at 18:22
  • $\begingroup$ Wondering if the Spirtes quote was before or after some of the newer software to come out implementing Pearl's notions. $\endgroup$ Oct 29, 2021 at 15:43
  • $\begingroup$ @AdrianKeister 1995? $\endgroup$
    – Alexis
    Oct 29, 2021 at 15:46
  • $\begingroup$ Yes, I think there have been some software releases since then that can automate some aspects of Pearl's framework. The Python dowhy library comes to mind, and I'm sure there are others. Perhaps this criticism of Spirtes is out-of-date? $\endgroup$ Oct 29, 2021 at 15:50
  • $\begingroup$ @AdrianKeister I do not see how you comments related to the substance of my answer, which is not about applying Pearl's framework, but about recognizing how ill suited it is for causal models represented by Directed Cyclic Graphs (as opposed to DAGs). Can you help me better understand? $\endgroup$
    – Alexis
    Oct 29, 2021 at 18:24
7
$\begingroup$

The most important criticism of Pearl's system is, from my perspective, that it has not yielded any practical, empirical advances anywhere it has been used. Given how long it has been around, there's no reason to think it will ever be a practical tool. This indicates that it can be used for some theoretical and perhaps didactic purposes, but a practical researcher will gain little from studying it.

$\endgroup$
6
  • 4
    $\begingroup$ Why is it ridiculous? If Pearl promoted his system simply as some kind of conceptual, philosophical tool for understanding what causality is, I wouldn't have a problem with it. But he constantly talks it up as a "revolutionary" practical tool for researchers to use, which is simply bullshit. For example, in his latest book Pearl says that he "would not be surprised" if the front-door method "eventually becomes a serious competitor to randomized controlled trials", which is a strong claim given that there's not a single example of the method being used to solve any real problem, ever. $\endgroup$
    – Matt
    May 22, 2019 at 8:27
  • 6
    $\begingroup$ Pearl's work has been cited tens of thousands of times. The front-door method was famously used to support the link between smoking and cancer in defiance of Ronald Fisher's testimony! $\endgroup$
    – Neil G
    May 22, 2019 at 8:47
  • 15
    $\begingroup$ What does Pearl's citation count have to do with anything? My criticism is that the practical benefits that he has promised for decades have not materialized. Pearl came up with the front-door criterion decades after Fisher had died and the cancer-and-smoking controversy had settled down. How could the criterion have been used against Fisher? $\endgroup$
    – Matt
    May 22, 2019 at 9:09
  • 4
    $\begingroup$ @Matt; I disagree with your point, I explain why in my answer. $\endgroup$
    – markowitz
    Dec 10, 2020 at 9:16
  • 3
    $\begingroup$ @Adrian Keister: What are those "real-world problems" that Pearl's framework solved? Smoking causing cancer doesn't apply because that became the consensus view long before Pearl's work on causality. $\endgroup$
    – Matt
    Jan 16, 2022 at 22:59
6
$\begingroup$

Reading answers and comments I feel the opportunity to add something.

The accepted answer, by rje42, is focused on DAG’s and non-parametric systems; strongly related concepts. Now, capabilities and limitations of these tools can be argued, however we have to say that linear SEMs are part of the Theory presented in Pearl manual (2000 or 2009). Pearl underscores limitations of linear systems but they are part of the Theory presented.

The comment of Paul seems crucial to me: “The whole point of Pearl's work is that different causal models can generate the same probabilistic model and hence the same-looking data. So it's not enough to have a probabilistic model, you have to base your analysis and causal interpretation on the full DAG to get reliable results.” Let me says that the last phrase can be rewritten as: so it's not enough to have a probabilistic model, you need structural causal equation/model (SCM). Pearl asked us to keep in mind a demarcation line between statistical and structural/causal concepts. The former alone can be never enough for proper causal inference, we need the latter too; here stays the root of most problems. In my opinion this clear distinction and his defense, represent the most important merit of Pearl.
Moreover Pearl suggests some tools such as: DAG, d-separation, do-operator, backdoor and front door criterion, among others. All of them are important, and express his theory, but all come from the demarcation line mentioned above, and all help us to work according with it. Put it differently, is not so tremendously relevant to argue pro or cons of one specific tool, it is rather about the necessity of the demarcation line. If the demarcation line disappears, all of Pearl's theory goes down, or, at best, adds just a bit of language to what we already have. However, this seems to me an unsustainable position. If some authors today still seriously argue so, please give me some reference about it.

I'm not yet expert enough to challenge the capability of all these tools, but they seem clear to me, and, until now, it seems to me that they work. I come from the econometric side and, about the tools therein, that I think the opposite. I can say that econometrics is: very widespread, very useful, very practical, very empirical, very challenging, very considered matters; and it has one of his most interesting topics in causality. In econometrics some causal issues can be fruitfully addressed with RCT tools. However, unfortunately, we can show that the econometrics literature, too often, addressed causal problems not properly. Shortly, this happened due to theoretical flaws. The dimensions of this problem emerge in their full width in:

Regression and Causation: A Critical Examination of Six Econometrics Textbooks - Chen and Pearl (2013) and

Trygve Haavelmo and the Emergence of causal calculus - Pearl; Econometric Theory (2015)

In these related discussions some point are addressed:

Under which assumptions a regression can be interpreted causally?

Difference Between Simultaneous Equation Model and Structural Equation Model

I don’t know if "equilibrium problems" invoked by Dimitriy V. Masterov cannot be addressed properly with Pearl SCMs, but from here:

Eight Myths About Causality and Structural Equation Models - Handbook of Causal Analysis for Social Research, Springer (2013)

it emerges that some frequently invoked limitations are false.

Finally, the argument of Matt seems to me not relevant, but not for "citations evidence" as argued by Neil G. In two words, Matt's point is

“Pearl's theory can be good for itself but not for the purpose of practice”.

This seems to me a logically wrong argument, definitely. Matter of fact is that Pearl presented a theory. So, it suffices to mention the motto “nothing can be more practical and useful than a good theory”. It is obvious that the examples in the book are as simple as possible, good didactic practice demands this. Making things more complicated is always possible and not difficult; on the other hand, proper simplifications are usually hard to make. The possibility to face simple problems or to rend them more simple seems to me strong evidence in favor of Pearl's Theory.

That said, the fact that no real issues are solved by Pearl's Theory (if it is true) is neither his responsibility not the responsibility of his theory. He himself complains that professors and researchers haven't spent time enough on his theory and tools (and on causal inference in general). This fact could be justified only in face of a clear theoretical flaw of Pearl's theory and clear superiority of another one. It is curious to see that probably the opposite is true; note that Pearl argued that RCT boil down to SCM. The problem that Matt underscores comes from professors' and researchers' responsibility.

I think that in the future Pearl's Theory will become common in econometrics too.

$\endgroup$
15
  • 4
    $\begingroup$ ”Matter of fact is that Pearl presented a Theory." What theorems did Pearl present? Is a 'do-operator' a theory? Does this theory make any change to the results when a researcher tests a physical theory at CERN or when a pharmacy company analyses the effectivity of a vaccine? Or does it just allow them to draw pretty diagrams like managers do? $\endgroup$ Dec 10, 2020 at 8:39
  • 2
    $\begingroup$ All things presented in Pearl 2009 represent a Theory, do-operator is relevant therein. This book is plenty of theorems, some referred on Pearl’s works. Graphs is only a language, concepts stay behind them. I’am not expert enough in Physics or pharmacy literature and/or practice for help you to find and discuss example about. $\endgroup$
    – markowitz
    Dec 10, 2020 at 9:15
  • 4
    $\begingroup$ But what sort of theory is it? A reference to an entire book does not help much in conveying what this theory is and is an alarm bell that there is no theory after all, and instead just a lot of mumbo jumbo. Is there any theory from Pearl that can be described by a few sentences or by a single formula? And is the theory a scientific theory, which means that it can be tested experimentally or proven mathematically? Or is it just a proposal for a different language, a set of definitions, expressions and methods? Or is it more like philosophy? $\endgroup$ Dec 10, 2020 at 9:48
  • 4
    $\begingroup$ However one “alarm bell” actually is here, it is your tone. Worse, in this discussion stats.stackexchange.com/questions/499455/… you confess: “I haven't read any of Pearl's books”, but anyway you feel the necessity to intervene here and speak against his main contribution. Forgive me for my frankness but your words reveal unjustified dislike towards Pearl and/or some ignorance. $\endgroup$
    – markowitz
    Dec 10, 2020 at 10:37
  • 2
    $\begingroup$ I read carefully what you written, as always, but is not easy for me visualize exhaustively you point of view; and comments are not for long discussions. You can add your answer if you think it can help. However I suppose that you not read carefully all what I written. Indeed above are given my replies on several question and doubts you posed me. $\endgroup$
    – markowitz
    Dec 10, 2020 at 17:17
-5
$\begingroup$

The synthesis behind causal reasoning is access to different aspects of reality/model using inferences from data. The paradoxical features in a model are avoided through formalisation in mathematical language. Informally, these counterintuitive features arise in observing a model unlike intervening through doing calculus.

The question on the openess of algorithms in the foundations of mathematics and logic is debated. One of the central point is that formalisations lead to harness infinite power to practice but turns these consistent procedures closed. In principle, while you do, you close the system unlike observing that has an active role in quantum mechanics unlike Pearl's causation hierarchy. Quantum contextuality asserts fundamental acausal relations between events, manifest as bugs! Bug could be a way into the depth of an iceberg! Technically, DAG are insufficient to reveal deeper structure of nature, where unrelated events could be related in a non-trivial way via topology.

Pearl's vision is significant in the sense that, he formally constructed causation theory and this integrity is elegant!

$\endgroup$
5
  • 6
    $\begingroup$ Can you point to a fully worked out practical example using observational data and taking into account data uncertainties, i.e., uncertainties of statistical inference? $\endgroup$ Oct 29, 2021 at 19:42
  • 4
    $\begingroup$ The conjoining of phrases like "quantum contextuality," "deeper structure of nature," and "topology" in a forum on statistics inevitably brings to mind the insights in Harry Frankfurt's book. $\endgroup$
    – whuber
    Oct 31, 2021 at 16:14
  • $\begingroup$ Causation is at the heart of algorithmic reasoning. Statistics is just a tool! Causal model is based on equivalence between the propositions that infer to an events and the events that actually took place. The impactful consequence of this indistinguishability is a subject matter of foundations of mathematics and computer science. Its not absurd! Quantum contextuality implies acasual/non-local relations between events that are interpreted to general classical systems using topology. @whuber $\endgroup$ Oct 31, 2021 at 23:34
  • $\begingroup$ See Budroni et al. 2021 for an introduction to the subject of quantum contextuality. $\endgroup$
    – Galen
    May 25, 2022 at 4:13
  • $\begingroup$ See Munkres 1999 for an introduction to topology. $\endgroup$
    – Galen
    May 25, 2022 at 4:15

Not the answer you're looking for? Browse other questions tagged or ask your own question.