Omitted variable bias: which predictors do I need to include, and why?

For a last couple of weeks I've been thinking about OVB (Omitted variable bias) in the context of regression and solution for that (how to avoid this problem). I am acquainted with Shalizi's lectures (2.2), but he is just describing this mathematically.

This week somebody said that it's quite easy - the solution for OVB is to include all those predictors that control the effect of confounding covariates, not all predictors for dependent variable Y.

I am not sure if this is true and yes, I do feel that I lack of deeper knowledge.

• Can you link to these lectures? – Matthew Drury Nov 9 '17 at 19:49

4 Answers

This is not necessarily wrong, but not always feasible and also not a free lunch.

An omitted variable may cause (see, e.g., the comments below for additional thoughts on the matter) bias if it is both (a) related to the outcome $Y$ and (b) correlated with the predictor $X$ whose effect on $Y$ you are primarily interested in.

Consider an example: You want to learn about the causal effect of additional schooling on later earnings. Another variable that is most certainly satisfies the conditions (a) and (b) is "motivation" - more motivated people will both be more successful in their jobs (whether they are highly schooled or not) and generally choose to receive more education, as they are likely to like learning, and not find it too painful to study for exams.

So, when comparing earnings of highly schooled and less schooled employees without controlling for motivation, you would likely at least partially not be comparing two groups that only differ in terms of their schooling (whose effect you are interested in) but also in terms of their motivation, so the observed difference in earnings should not only be ascribed to differences in schooling.

Now, it would indeed be a solution to control for motivation by including it into the regression. The likely problem is of course: are you going to have data on motivation? Even if you were to conduct a survey yourself (rather than use say administrative data, that will most likely not have entries on motivation), how would you even measure it?

As to why including everything is not a free lunch: if you have a small sample, including all available covariates may quickly lead to overfitting when prediction is your goal. See for example this very nice discussion.

• Cristoph, you might want to be a bit more precise regarding your second paragraph --- you can find some counterexamples for this definition of bias here: stats.stackexchange.com/questions/59369/confounder-definition/… – Carlos Cinelli Nov 9 '17 at 17:54
• Your definition does not discredit mediating variables or colliders for adjustment; adjusting for these variables cause biases of collider bias or effect attenuation. It also operates under a closed world example, in this you must assume you have measured all possible confounding variables, an assumption which is rarely met, nor discussed. A full definition of confounding bias is rather difficult. – AdamO Nov 9 '17 at 18:02
• Further, In inferential statistics, it is important to label "motivation" as a confounding variable per the earlier discussions. Also, your comments only apply to GLMs with linear or log links. – AdamO Nov 15 '17 at 20:30

the solution for OVB is to include all those predictors that control the effect of confounding covariates not all predictors for dependent variable Y.

Yes, this is correct if you are more precise about it. For identification purposes, you should include the variables that control the effect of confounding and avoid those that open confounding paths or mediate the effect you are trying to measure (if you are interested in the total effect) --- that is, you should include those variables that satisfy the backdoor criterion. You should not indiscriminately include all predictors of $$Y$$, if by predictor you mean anything that "predicts" $$Y$$ --- this could bias your estimate. You can find a gentle, example-based, introduction to the topic in this Crash Course in Good and Bad Controls.

In this same vein, it's worth noticing that Christoph's answer is not strictly correct:

an omitted variable causes bias if it is both (a) related to the outcome Y and (b) correlated with the predictor X whose effect on Y you are primarily interested in

This is not true. Correlational criteria is not necessary nor sufficient to define what a confounder is. This is a common misconception on the definition of confounders, illustrated in this other answer.

Of course, which variables to include that guarantees identification addresses only the matter of getting consistent estimates of the causal quantity of interest. You have many other problems to address, such as the efficiency of your estimate (so you might choose/avoid variables that reduce/increase variance), biases due to misspecification of the functional form etc.

• (+1) can you provide a lay-explanation of what the backdoor criterion is? – AdamO Nov 9 '17 at 18:07
• @AdamO basically a set of variables that blocks the effect of common causes, that do not open new confounding paths (such as colliders, but you in some cases as in fig 3.4 of the pdf you might need to control for colliders, so you need to further block the opened path) and that do not include variables that mediate the effect you are trying to measure (if you are interested in the total effect). – Carlos Cinelli Nov 9 '17 at 18:09
• Indeed, the sake of the request was to increase the accessibility of your (very good) answer. – AdamO Nov 9 '17 at 18:31
• You precised that backdoor criterion deal with “total effect”. Now, in linear SCM, is correct to say that if our SCM is fully specified by only one structural equation any direct effect coincide with total? Then, no indirect effect exist? And any single structural parameter have total effect meaning? If the reply is too articulate I can open another question – markowitz Sep 18 '19 at 14:23
• @markowitz it’s not correct, you need to explicitly say that all other variables do not cause each other—only then, by assumption you are saying there’s no indirect effect. – Carlos Cinelli Sep 20 '19 at 4:09

Theoretically, including all relevant predictors eliminates the omitted variable bias. However, it might not always be feasible to include all relevant explanatory variables in your regression (due to unawareness of relevant variables or lack of data).

Regarding the lack of knowledge about the omitted variable bias. There are a couple of good lectures out there on the OVB. Looking around, one of the most comprehensive lectures on the omitted variable bias might be this one:

https://economictheoryblog.com/2018/05/04/omitted-variable-bias

It also includes a section that discusses possible strategies against an omitted variable bias.

• Including all variables can hurt the prediction accuracy of the model. It also can create a multi-collinearity problem . I am not sure what you mean when you say "it might not always be feasible to include all possible explanatory variables". – Michael R. Chernick May 7 '18 at 5:40
• Sandro, this is false and the problem is even beyond accuracy. Including some predictors will cause permanent bias to your estimate, even with infinite samples, see stats.stackexchange.com/questions/59369/confounder-definition/…. – Carlos Cinelli May 7 '18 at 5:50
• You are right. I meant all relevant explanatory variables. I adjusted my answer. Thanks. – Sandro Salter May 7 '18 at 6:03
• @SandroSalter what do you mean by relevant? You need to be precise here. – Carlos Cinelli May 7 '18 at 6:08

Carlos' answer is good in that it addresses a major deficiency in regression modeling practice. The term OVB is very imprecise. Except under atypical mathematical structures, adjusting for other variables will change the effect estimated for a primary regressor. This alone does not mean all such variables should be included in a model.

The "backdoor criterion" specifically addresses confounding bias. An audience of experts will generally not accept/believe results from models which omit confounding variables from adjustment. This is for good reason. Omitted confounders have led to completely incorrect inference in large confirmatory studies, and further led to policies, drug indications, or media coverage which were costly and damaging. The preferred terminology here is confounding bias, rather than merely OVB. This applies to all types of models, including the most prevalent linear regression.

The second most prevalent (perhaps) model is logistic regression. There is another type of "bias" (perhaps) which arises from logistic models unrelated to confounding. You can change the primary effect by adjusting for variables which are uncorrelated with the primary regressor. This is because of the non-collapsibility of the odds ratio. This arises when the primary exposure has a heterogeneous distribution of covariates underlying the baseline risk of the outcome. The slope of the sigmoid which estimates the "averaged out" accumulation of risk per unit difference in a primary regressor is attenuated. This type of bias arises when the target of inference was individual level risk, rather than population averaged.

In general, the advice to modelers is to adjust for prognostic variables, or variables which, despite being unrelated to the primary regressor, are causally predictive of the outcome. Examples might be in a study on lung cancer and smoking, groups of participants by environmental ambient pollution. Assume for the moment no evidence suggests that differences in regionality satisfies backdoor criteria to confound the smoking-cancer relationship. However, the difference in risk for this environmental exposure substantially predicts risk of lung cancer. Adjusting for environmental exposure more finely stratifies these participants so that the apparent differences between smoking and non-smoking, and cancer risk are apparent.

A very nice description of the difference is found here: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3147074/pdf/dyr041.pdf