19
$\begingroup$

I am trying to understand the point of the second panel in the following xkcd comic:

https://t.co/EfTKCyvCFS

Specifically, how can one be misled by controlling too many confounding variables in one's models?

Any pointers to what this criticism is called in the literature—so I can look into it further—will be appreciated.

$\endgroup$
3
  • 1
    $\begingroup$ I wrote a lay explanation here $\endgroup$
    – Barmar
    Jan 18 at 16:07
  • 2
    $\begingroup$ See Professor Pearl's A Crash Course in Good and Bad Control $\endgroup$
    – msuzen
    Jan 19 at 17:38
  • 2
    $\begingroup$ Just to give full credit to all the authors: The article in the link is written by Carlos Cinelli (a frequent contributor on this site), Andrew Forney, and Judea Pearl. $\endgroup$ Feb 2 at 11:44

5 Answers 5

23
$\begingroup$

There is no such thing as a "sweet spot" for the number of variables to control for in order to get an unbiased estimate of the causal effect. Since we are talking about confounding, we must have in mind the estimation of the causal effect of a particular variable. You use a graphic tool called the DAG to map out the causal relationships and then you condition on a set of variables that will yield you the causal effect. Conditioning on variables generally blocks the flow of association but conditioning on a collider (common effect) will induce association between variables that are not causally related. The more variables you condition on, the more likely you are to condition on a collider and thus induce association without causation; that said the more variables you condition on you are also blocking more backdoor paths, including those with colliders. The reasoning here should not revolve around "how many variables?" but around "which variables?" to condition on.

Below is an example where not conditioning on anything is what you want in order to estimate the direct causal effect of A on B. On the other hand, conditioning on the set {D} or {C,D} will bias the direct causal effect of A on B because it conditions on the collider D and opens backdoor path(s).

enter image description here

This post here can serve as a good introduction to causal reasoning with DAGs.

$\endgroup$
12
$\begingroup$

I would point out three things:

(1) Generally (related to the estimation of causal effects)

Usually you want to explain phenomena out there in the world with parsimonious models including variables deduced from some theory. You may just add any variable that comes to your mind to a regression model and end up with an almost perfect fit, but you did not learn anything about (or even fundamentally distorted) the relationship (aka causal/treatment effects) you actually interested in (also see the DAGs @ColorStatistics pointed to). (Literature e.g.: "Causal Inference in Statistics" by Judea Pearl).

(2) Specifically (more related to the overspecified model term)

You can perceive adding irrelevant variables to a regression model as estimating coefficients on irrelevant variables that are truly zero. Then if you do this the estimators of our regression coefficients are still unbiased, but also inefficient since we did not consider (true) zero restrictions on the coefficients of the irrelevant variables. Hence, inference stays valid but confidence intervals become broader. (Literature: basically any econometrics textbook, e.g. Wooldrige).

(3) Additionally (related to prediction)

If you are solely interested in prediction performance of a model based on your training data then adding 'irrelevant' variables to your model is less harmful (irrelevant in the sense of being not causal and not in the sense of having true zero restrictions on the coefficients). As the overspefication of your model only becomes problematic if you want to do inference (broader confidence intervals). (Have a look in the causal machine learning literature).

$\endgroup$
2
  • 5
    $\begingroup$ Relating to point (3): have a look at models 8 and 9 in A crash course in good and bad controls, and the cited literature there. $\endgroup$ Jan 18 at 7:43
  • 1
    $\begingroup$ @timm: with regards to point 2 the suggested link is a very good one; you'll see there that sometimes controlling for variables that you'd call irrelevant because they are bias-neutral actually improves precision. An example is controlling for Z in trying to estimate the causal effect of X on Y in $X \rightarrow Y \leftarrow Z$. $\endgroup$ Jan 20 at 18:53
5
$\begingroup$

Well, it's related to the concept of p-hacking. Given a sufficient number of plausible confounding variables to be added in the study, it's possible to find a combination of them that would yield significant results (you just plug in or out until you get significant results, and you report those).

There's a very nice post in FiveThirtyEight about this so you can experience the idea, where you can even obtain contradictory results depending on which variables you wish to "correct for".

$\endgroup$
1
$\begingroup$

The term you are looking for is overfitting. Wikipedia has a good explanation.

$\endgroup$
5
  • 8
    $\begingroup$ I don't think so. Overfitting makes sense only in a predictive context. Here the context is clearly causal. We are not trying to explain the variability in the dependent variable here; nor are we trying to model the DGP, so it makes little sense to speak of overfitting here; our goal is singular - to obtain an unbiased estimate of the causal effect of variable A on variable B. You can add as many variables in the regression as you'd like, but you will in no sense overfit the causal effect. That we'll overfit the DGP is a correct answer to a question we don't care to ask in this exercise. $\endgroup$ Jan 17 at 15:28
  • 2
    $\begingroup$ @ColorStatistics Agreed that overfitting used in predictive capacity sense in general and "overfit the causal effect" is an ambgious term here. But in general, I wouldn't dismiss the concept of overfitting in causal inference. Overfitting is a much more generic concept, philosophically. It is about "complexity measure" and "Occam's razor" in general and could be raised for "bad controls" on DAGs. DGPs are naturally connected if used in conjunction with DAGs. See also Gelman's What is overfitting? $\endgroup$
    – msuzen
    Jan 19 at 23:13
  • $\begingroup$ The OP showed a cartoon (so limited context) and asked for the term that can be looked up to learn about the problem of including too many independent variables. I answered with "overfitting". As of 21Jan2022 AM, five people upvoted, but four downvoted this answer. @ColorStatistics added a great alternative answer about DAG which I learned from and upvoted. But I don't understand the downvotes. I don't care about my vote counts, but I do want to understand why the "overfitting" answer is considered not just to be incomplete, but wrong enough to be downvoted. ?? $\endgroup$ Jan 21 at 15:26
  • $\begingroup$ @HarveyMotulsky: I cannot speak for the downvotes, but here's the paragraph that I think best embodies the message: "Regression should be taught the way it is now most often used: as a tool to control for confounding factors. This approach abandons the traditional regression framework in which all regressors are treated equally. The pedagogical emphasis on statistical efficiency and functional form, along with the sophomoric narrative that sets students off in search of “true models” as defined by a seemingly precise statistical fit, is ready for retirement. $\endgroup$ Jan 21 at 16:17
  • $\begingroup$ @HarveyMotulsky: Instead, the focus should be on the set of control variables needed to insure that the regression-estimated effect of the economic variable of interest has a causal interpretation." This is from Angrist and Pischke and their advice on how econometrics should be taught. $\endgroup$ Jan 21 at 16:17
1
$\begingroup$

There are some helpful mathsy explanations, but I thought perhaps this could use an intuitive example.

Suppose that you're investigating (perhaps for an insurance company) whether hair colour has an impact on crash risk. You look at the data, and at first pass you see that brunettes are 10% more likely to crash than blondes. But in the same data you see that brunettes are also more likely to get caught speeding. You do the controls to take out the effect of speeding, you'd find that the effect of hair on crash risk drops below signifance.

That would probably be an example of an inappropriate thing to control. It is likely that the fact that our brunettes speed more is the mechanism by which they're more likely to crash. As such, if you insist on zeroing out that mechanism, you're forcing yourself to see no effect even if it's obviously there. Intuitively, "Actually brunettes are very safe drivers considering how much they speed" is very obviously an unreasonable defence to make!

Conversely, suppose we look at the dataset again and find that people who are bald are 50% more likely to crash than those with red hair. But it also happens that the bald people in the dataset were typically older men, and younger women were underrepresented. Again you throw statistical controls at the situation and the effect disappears. This is probably a good thing to control, not least because your insurance company already asks about age and gender so you don't want to double count any effects. Again intuitively, saying "We already knew that age and gender are known to have an impact on road safety and on baldness prevalence. These data show bald young women are just as safe as hairy young women and bald old men are just as safe as hairy old men." seems like a very reasonable clarification.

(This example is entirely made up, and isn't an allegation that any particular hair type in the real world are dangerous drivers!)

$\endgroup$
1
  • $\begingroup$ (The bad example is an example of the simplest, Model 11, in the crash course document linked in several comments.) $\endgroup$
    – Josiah
    Jan 20 at 18:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.