How would econometricians answer the objections and recommendations raised by Chen and Pearl (2013)?

Question

In their article, Chen and Pearl (2013), critically examined 6 econometric textbooks, among these the textbooks written by Wooldridge (2009) {the introductory book}, and Stock & Watson (2011). These last 2 books are either the core textbook or at the top of the list of suggested readings for many introductory econometrics courses out there. I think that somebody just about to start their introductory econometrics class using one of these books would be well served to know which material in their textbook is contested as potentially incomplete/unclear/misleading (as Chen and Pearl argued) and if it is not incomplete/unclear/misleading then a response to these accusations would give the student renewed confidence that his/her textbook will not lead him/her astray, and will alert the student to the complete/accurate picture.

Ideally the response to Chen and Pearl would come from these 6 (sets of) authors themselves. Did they respond? In the 7 years since the article, did any of the 6 (sets of) authors make any changes to their books which are in line with the recommendations by Chen & Pearl (2013)? For example, I see that Wooldridge released a new version of his introductory textbook in 2019; in it did he make any changes in line with Chen & Pearl's recommendations? More broadly, through this imperfect device I am trying to get a sense for whether econometricians, as a whole, found anything useful in Chen & Pearl's critical review and made any changes/improvements as a result.

Absent a response from the 6 (sets of) authors themselves, a response by the econometricians/statisticians on this forum might be the next best thing. My hope is that people can respond to the issues raised by Chen & Pearl (repeated by me below) and hopefully a consensus view will emerge that can be of use to students using these textbooks. Below, I restrict the scope to 2 of the analysis to Wooldridge (2009) and Stock & Watson (2011) and only to the questions where Chen & Pearl objected with either of these books (see table below with disagreements highlighted).

Q2. Does the author present example problems that require prediction alone?

Wooldridge does not present any examples that require prediction alone.

Q4. Does the author define $\boldsymbol{\beta}$ by the equality, $\boldsymbol{\beta X = E[Y|X]}$ ?

"... Haavelmo shows that $\beta X$ is not equal to the conditional expectation, but rather to the expected value of $Y$ given that we intervene and set the value of $X$ to $x$. This "intervention-based expectation" was later given the notation $E[Y|\text{do}(x)]$ in (Pearl, 1995)" (page 2 of the Chen & Pearl article).

“The first part of Equation (4.5), $\beta_0 + \beta_1 X_{i}$, is the population regression line or the population regression function. This is the relationship that holds between $Y$ and $X$ on average over the population. Thus, if you knew the value of $X$, according to this population regression line you would predict that the value of the dependent variable, $Y$ , is $\beta_0 + \beta_1 X$.”(Stock and Watson, 2011, p. 110)" (page 20 of the Chen & Pearl article).

My commentary: Strictly speaking, both authors define $\beta$ by the equality, $\beta X = E[Y|X]$, however Wooldridge does so in the context of a randomized experiment (randomizing fertilizer to be applied to plots) and he gets a pass, while Stock & Watson do so in the context of an observational study and they get flagged. If I understand it correctly, Chen & Pearl correctly interpret the context of Wooldridge's model as causal and they see $\beta X = E[Y|do(X)]$ where Wooldridge writes $\beta X = E[Y|X]$. However in the Stock & Watson case they deem the context of the model to be causal (which seems questionable, see below) and thus flag them for writing $\beta X = E[Y|X]$.

Stock and Watson motivate their simple regression model with the following statement. "If she reduces the average class size by two students, what will the effect be on standardized test scores in her district?" There also is no "ceteris paribus" statement, which in my mind suggests that the model is intended to be merely predictive rather than causal.

In Chen & Pearl's defense, Stock & Watson's introduction to this chapter "Linear Regression with One Regressor" is unclear, as they dance around with both casual and predictive contexts. They start by asking 3 causal sounding questions and then state that "Just as the mean of $Y$ is an unknown characteristic of the population distribution of $Y$, the slope of the line relating $X$ and $Y$ is an unknown characteristic of the population joint distribution of $X$ and $Y$". (Stock & Watson, page 155)

Q7. Does the author state that each structural equation in the econometric model is meant to capture a ceteris paribus or "everything else held fixed" relationship?

My commentary: It seems to me that Stock & Watson introduced both simple linear regression and multiple linear regression in a predictive context, which is why they did not mention ceteris paribus. Is that accurate?

Q8. Does the author assume that exogeneity of $\mathbf{X}$ is inherent to the model?

My commentary: I don't fully understand this point but to me this seems like a potentially fair criticism of Wooldridge's exposition. His entire book has a causal context and yet he states that "The crucial assumption is that the average value of u does not depend on the value of $x$". (page 25) However, on the very next page he says "In the fertilizer example, if fertilizer amounts are chosen independently of other features of the plots, then (2.6) $\{E(u|x)=E(u)\}$ will hold: the average land quality will not depend on the amount of fertilizer. Seems like a flaw to call it the crucial assumption without further clarifying in what contexts it is crucial and in what contexts it is trivial. Hopefully someone can clarify and correct me if I am wrong.

Q10. Does the author use separate notation for $\boldsymbol{E[Y |\textbf{do}(X)]}$ and $\boldsymbol{E[Y |X]}$?

Q11. Does the author use separate notation for the slope of the line associated with $\boldsymbol{E[Y |X]}$ and that associated with $\boldsymbol{E[Y |\textbf{do}(X)]}$?

These last 2 are about notation, and while I think Pearl effectively pointed out the problems stemming from the lack of something like this notation $E[Y |\text{do}(X)]$, I am not sure there is agreement yet on whether this is the way to go, so would seem unfair to flag any econometrics book for not using this notation. Is [Gelman] saying he is against such notation? "Setting aside any problems I have with such models (I don’t actually think the “do operator” makes sense as a general construct, for reasons we’ve discussed in various places on this blog from time to time), the point is that these are qualitative, on/off statements. They’re “if-then” statements, not “how much” statements."

Absent responses from these authors to Chen & Pearl and given how widely used these textbooks are, I think this thread could be of great interest to an army of future econometricians/statisticians/data analysts that are currently studying from these textbooks.

Carlos Cinelli's answer here could help provide further context.

Answer 1

Absent a response from the 6 (sets of) authors themselves, a response by the econometricians/statisticians on this forum might be the next best thing. My hope is that people can respond to the issues raised by Chen & Pearl (repeated by me below) and hopefully a consensus view will emerge that can be of use to students using these textbooks.

I can propose you my perspective about this great point. My analysis is not complete yet but most conclusions are outlined. I can defend what I will say, even if I don't have time and space enough here. Naturally I can go wrong, after all I'm not a Professor. If my points are wrong forgive me. I stay here for read more opinion too, and learn something about that.

I started to faced the problem of causality in econometrics some years ago and, also before to read Chen and Pearl (2013), it seemed me that some problems appeared.

I surveyed several econometrics manuals, all six considered in Chen and Pearl (2013) and several others. Moreover I studied many articles, slide and related material in general. My conclusion is that, too often, causal questions was not properly addressed in econometric literature.

The story can be very long but we can start to noting that: is hard to find two manuals that share exactly the same assumptions and/or implications, this factor affects primarily causal questions. Obviously most concepts are shared but some differences are relevant and not always can be easily solved. Anyway, keeping aside specific points/comparisons, is hard to find among econometrics books the exact set of assumptions that justifies causal interpretation for regression. This is quite puzzling because causal questions are, or should be, tremendously important in econometrics. Initially I thought that the problems I encountered, if any, boiled down in some details. Over time the situation seemed more serious to me. When I encountered Chen and Pearl (2013) my perplexities was confirmed and increased and go deeper. Time ago I considered simply not possible that several econometric Masters made so serious mistakes. Today I’m convinced that, surprisingly, things is so. Therefore most generalistic econometric books should be revised; in some case completely rewritten.

It seems me that all problems come from conflations between causal and statistical concepts, as used in most econometric manuals and literature in general.

Indeed, statistical concepts and assumptions must be clearly separated from causal one. All statistical assumptions can be considered as restrictions on some joint probability distributions, but joint probability distributions cannot encode causal assumptions.

In my view clear position about that and related proposed remedies represent the most important contribution of Pearl’s literature. Several people critic his works, but in my opinion most issues are bad posed. Read here for my perspective about that: Criticism of Pearl's theory of causality

Several of my questions and answers on this site swing around “regression and causation” and most things are summarized here:

Under which assumptions a regression can be interpreted causally?

Most conflation problems swing around the controversial concepts of exogeneity, error terms and true model (see more below).

Now, the problems are far from uniformly distributed among materials and, presumably, among peoples understandings. However, in some extent, the problems are widely shared and reveal that, at general level, both type of concepts, or at least the causal ones, are badly understood. As consequence, in short, we can says that, in general, the “current Econometric Theory about causality” is flawed.

Note that the problems underscored in Chen and Pearl (2013) do not stay only there but emerged also elsewhere. One relevant article about it is:

Trygve Haavelmo and the Emergence of causal calculus - Pearl; Econometric Theory (2015)

Some econometricians replied to Pearl:

CAUSAL ANALYSIS AFTER HAAVELMO - Heckman and Pinto

and Pearl replied to them

Reflections on Heckman and Pinto's “Causal Analysis After Haavelmo" – Pearl

Moreover two of most eminent econometricians focused of causal part of econometrics are Angrist and Pischke. Indeed in my opinion, and not only, the best econometrics book about causality is their: Mostly Harmless Econometrics: An Empiricist's Companion - Angrist and Pischke (2009). Them can be considered as eminent Authors of “experimental school”.

Relevant to see that Angrist and Pischke too are critics about how Econometrics, and his causal part in particular, is teached. See here: Undergraduate Econometrics Instruction: Through Our Classes, Darkly - Journal of Economic Perspectives—Volume 31, Number 2—Spring 2017—Pages 125–144

The core of problems, not by chance, swing around error terms and, then, exogeneity:

For the most part, legacy texts have a uniform structure: they begin by introducing a linear model for an economic outcome variable, followed closely by stating that the error term is assumed to be either mean-independent of, or uncorrelated with, regressors. The purpose of this model—whether it is a causal relationship in the sense of describing the consequences of regressor manipulation, a statistical forecasting tool, or a parameterized conditional expectation function—is usually unclear. Pag 138

Some problems underscored in Chen and Pearl (2013) are avoided in Angrist and Pischke (2009), but no Pearl's tools are used or suggested. Indeed Pearl is critic about them too as we can see here:

https://p-hunermund.com/2017/02/22/judea-pearl-on-angrist-and-pischke/

… the debate about causality in econometrics seems far from close.

Focusing on your specific points:

Ideally the response to Chen and Pearl would come from these 6 (sets of) authors themselves. Did they respond? In the 7 years since the article, did any of the 6 (sets of) authors make any changes to their books which are in line with the recommendations by Chen & Pearl (2013)?

I don't know if some private replies has been given. However the article of Chen and Pearl is public, therefore public should be the replies. At the best of my knowledge no public reply exist yet. Obviously this is not good for a defence of econometric manuals. Me too are waiting for public reply of Authors.

What I can says you is that for several manuals involved, new versions has been released. For example in Greene 8th edition (2018) the critics and suggestions of Chen and Pearl (2013), that analyze Green 7th edition (2012), are completely not considered; no words are spent in the direction suggested by Chen and Pearl. For Wooldridge the analyzed version is the 4th edition (2009); others has been released, today we are at 7th edition (note that the publication date can depend on the translation or others detail but the edition number should be always consistent). As in Greene's manual case, suggestions are not considered at all. For Stock and Watson the analyzed version is the 3rd and the 4th has been released recently. Interesting to note that is SW case some detail about machine learning topics was added and for casual concepts some things was modified. However these adding/modifications seems follow the Angrist and Pischke suggestions more than the Pearl ones; note that the name of Pearl appear in the acknowledgment (in the others books Pearl is not cited at all). At the best of my knowledge no one econometric book has yet take seriously Pearl’s suggestions.

Said that, I’m grateful to Chen and Pearl for their article, however I do not appreciate so much the table used there. All therein concepts can be mixed and, worse, all related problems can be easily masked if considered separately, as any table suggest. My analysis is not complete yet but I think that we have to give at Authors the possibility to explain exhaustively what they mean and without force them to use Pearl language and tools. Therefore I do not consider points like Q10 and Q11. If we consider the Pearl language and tools as mandatory the analysis is easy and fast and bring us to conclude: no one econometrics book can be saved. Indeed this seems the Pearl opinion, I heard it by he in a published lesson on Youtube.

I’m open mind with econometrics Authors strategy, but them must demonstrate the consistency of their arguments. I think that eventual inconsistency of the theory presented from Authors can be revealed from incorrect/contradictory/ambiguous statements along the books. For this scrutiny, semantics and examples matters a lot. In Chen and Pearl (2013) some very important points are reported, I follow the same idea but giving more time to the Authors. Anyway, as already said before, It seems me that the argument spent in most manuals are inconsistent about causality. I share most points and conclusions of Chen and Pearl (2013), maybe something more should be added.

Now some comments more:

Q2. Does the author present example problems that require prediction alone? Wooldridge does not present any examples that require prediction alone.

This question give me the possibility to show briefly what I intend with “my analysis”. I consider here 7th edition, the last. What is sure is that prediction and causal scope for an econometric model is not clearly separated and well treated. Let me report some parts of this book.

The distinction between causation and prediction seems recognized:

Even when economic theories are not most naturally described in terms of causality, they often have predictions that can be tested using econometric methods. The following example demonstrates this approach. Pag 14

However the distinction is not clearly transposed in the assumptions and, then, in the econometric theory presented.

Indeed we have:

MLR.1: Introduce the population model and interpret the population parameters (which we hope to estimate). … MLR.4: Assume that, in the population, the mean of the unobservable error does not depend on the values of the explanatory variables; this is the “mean independence” assumption combined with a zero population mean for the error, and it is the key assumption that delivers unbiasedness of OLS. From Preface

Note that that both unobservable and population concepts are used in MRL.4

Moreover is said that MLR.1 is about "linear parameter population model" (true model) pag 80.

Moreover is added

When Assumption MLR.4 holds, we often say that we have exogenous explanatory variables. Pag 82

Later is introduced an important paragraph “Several Scenarios for Applying Multiple Regression”. There (pag 98/101) It is said that

MLR.4 is true by construction once we assume linearity. Pag 98

this fact is strange because mean that MLR.1 imply MRL.4; redundant assumption. Worse, the fact that MLR.4 hold by construction under linearity exclude all possibility to consider the error term, then the true model in general, as something like structural/causal. Said that, at this point, MLR.4/MLR.1 are used for justifies that linear regression estimated with OLS is good for prediction.

Moreover

Multiple regression can be used to test the efficient markets hypothesis because MLR.4 holds by construction once we assume a linear model

The redundant argument go ahead, and MLR.4 is used here to demonstrate that linear regression estimated with OLS is good for testing economic theory, a causal concept. Honestly we can argue here that, even if come from economic theory, efficient markets hypothesis can be consider a predictive more than causal concept. However in this case we can ask: why this example is presented in a different scenario from prediction?

Later other two “scenarios” for regression are presented: Testing for Ceteris Paribus Group Differences and Potential Outcomes, Treatment Effects, and Policy Analysis

So all these seems presented as notably different scopes (scenarios). Moreover the argument

… and so MLR.4 holds by construction. OLS can be used to obtain an unbiased estimator of $\beta$ (and the other coefficients).

is used for the former case. For the latter [pag 100/101] an ad hoc conditional independence assumption is introduced; no mention for MRL.1 to MRL.4.

Later at the paragraph “Revisiting Causal Effects and Policy” pag 151, it is said

In Section 3-7e [pag 100/101] we showed how multiple regression can be used to obtain unbiased estimators of causal, or treatment, effects in the context of policy interventions … We know that the OLS estimator of $\tau$ [treatment effect] is unbiased because MLR.1 and MLR.4 hold (and we have random sampling from the population [=MRL.2])

then … even if before was not considered … now the assumptions MRL.1 to MRL.4 are enough for causation … even if them are pure statistical assumptions … and even if them are the same good for prediction … even if prediction and causation are different goals

Moreover the structural equation concept is introduced much after, at pag 505 in the context of IV and 2SLS estimator (Chapter 15)

We call this a structural equation to emphasize that we are interested in the $\beta_j$, which simply means that the equation is supposed to measure a causal relationship

also concepts like reduced form and identification are introduced there. Those concepts are used again in Chapter 16 that is about simultaneous equation models.

This strategy give the impression that structural concepts, and related ones, are a special subject, tolerably different from what exposed before. But if so, why causality is used and justified also before? Why only now structural and related concepts are needed?

Moreover, in the introduction of Chapter 16 is said that all remedies presented in the book deal with endogeneity problem; used in the same chapter as causal concept. Among others, this give the impression that endogeneity is the core problem for any scenarios/scope, prediction included. Worse, before is affirmed that under linearity MLR.4 hold, then exogeneity hold, then endogeneity go away, then causal conclusions are permitted by construction.

All this story, and so this book in general, seems me strongly problematic. Books like this bring the readers in insurmountable confusions.

Today I’m convinced that the bad treatment of causality go together with not so good treatment of prediction. In some extent this seems me true for most generalistic econometric books. For this reason too, most of them should be revised if not completely rewritten. Read also here: What is the relationship between minimizing prediciton error versus parameter estimation error?

these points are usually not well recognized in most generalistic econometric manuals.

Now, you ask

Stock and Watson motivate their simple regression model with the following statement. "If she reduces the average class size by two students, what will the effect be on standardized test scores in her district?" There also is no "ceteris paribus" statement, which in my mind suggests that the model is intended to be merely predictive rather than causal. … My commentary: It seems to me that Stock & Watson introduced both simple linear regression and multiple linear regression in a predictive context, which is why they did not mention ceteris paribus. Is that accurate?

No. Stock and Watson spent most of their time to speak about causality, and them recognize tolerably well the distinction between forecasting and causality (read paragraph 9.3). Even in Chen and Pearl (2013) this fact is underscored. Luckily I faced econometrics for the first time with SW manual. Class size effect on standardized test score results is used precisely as clear causal question. Staying at 3th edition, assumptions presented in the Chapter 4 to 9, the core of the book, must be intended as justifications for causal interpretation of regression. Them follow the as if experimental paradigm. Less time is spent for pure prediction, mainly Chapter 14. Unfortunately SW conflate causal and statistical concepts and seems that causal conclusions is founded on statistical assumptions.

Q8. Does the author assume that exogeneity of X is inherent to the model?

As said before most problems swing around exogeneity assumption. All six books in argument use exogeneity, in some form considered, as crucial assumption. However no one of them use exogeneity concept properly. This remain true for most econometrics books. Them use exogeneity ambiguously, like a concept that flip from statistical to causal side and/or mix them.

I started my analysis precisely since exogeneity assumption problems. Today I have no doubt more about that. Pearl is right about exogeneity, it is a causal concept. Only if we accept this perspective, and work consistently with that, all ambiguities and contradictions come to solve.

Read also here:

Zero conditional expectation of error in OLS regression

Does homoscedasticity imply that the regressor variables and the errors are uncorrelated?

Multiple Linear Regression Zero Conditional Mean Assumption

Another great and related problem swing around the concept of true model. In most econometric books it is erroneously conflated with something like population regression or, worse, some ambiguous statistical object. If we take the so called true model as a structural linear causal model, all problems come to solve. Read also here:

Regression and the CEF

What is a 'true' model?

linear causal model

Answer 2

Different subjects indeed treat causality differently. It will largely affect the coherence of an econometrics textbook if thoroughly adopting the statistical causality. Actually economists reveal the causality, including how to specify a regression equation, both in structural form or reduced form (i.e., statistical causality) empirically. Hansen from UWM did a comprehensive job for both methodologies, and Joshua Angrist wrote famous MHE for the reduced form.