Do we really need to include "all relevant predictors?"

Question

A basic assumption of using regression models for inference is that "all relevant predictors" have been included in the prediction equation. The rationale is that failure to include an important real-world factor leads to biased coefficients and thus inaccurate inferences (i.e, omitted variable bias).

But in research practice, I have never seen anyone including anything resembling "all relevant predictors." Many phenomena have a myriad of important causes, and it would be very difficult, if not impossible, to include them all. An off-the-cuff example is modeling depression as an outcome: No one has built anything close to a model that includes "all relevant variables": e.g., parental history, personality traits, social support, income, their interactions, etc., etc...

Moreover, fitting such a complex model would lead to highly unstable estimates unless there were very large sample sizes.

My question is very simple: Is the assumption/advice to "include all relevant predictors" just something that we "say" but never actually mean? If not, then why do we give it as actual modeling advice?

And does this mean that most coefficients are probably misleading? (e.g., a study on personality factors and depression that uses only several predictors). In other words, how big of a problem is this for the conclusions of our sciences?

Answer 1

You are right - we are seldom realistic in saying "all relevant predictors". In practice we can be satisfied with including predictors that explain the major sources of variation in $Y$. In the special case of drawing inference about a risk factor or treatment in an observational study, this is seldom good enough. For that, adjustment for confounding needs to be highly agressive, including variables that might be related to outcome and might be related to treatment choice or to the risk factor you are trying to publicize.

It is interested that with the normal linear model, omitted covariates, especially if orthogonal to included covariates, can be thought of as just enlarging the error term. In nonlinear models (logistic, Cox, many others) omission of variables can bias the effects of all the variables included in the model (due to non-collapsibility of the odds ratio, for example).

Answer 2

Yes, you must include all "relevant variables", but you must be smart about it. You must think of the ways to construct the experiments that would isolate the impact of your phenomenon from unrelated stuff, which is a plenty in real world (as opposed to a class room) research. Before you get into statistics, you have to do the heavy lifting in your domain, not in statistics.

I encourage you not be cynical about including all relevant variables, because it's not only a noble goal but also because it's often possible. We don't say this just for the sake of saying it. We really do mean it. In fact, designing experiments and studies that are able to include all relevant variables is what makes science really interesting, and different from mechanical boiler plate "experiments".

To motivate my statement, I'll give you an example of how Galileo studied acceleration. Here's his description of an actual experiment (from this web page):

A piece of wooden moulding or scantling, about 12 cubits long, half a cubit wide, and three finger-breadths thick, was taken; on its edge was cut a channel a little more than one finger in breadth; having made this groove very straight, smooth, and polished, and having lined it with parchment, also as smooth and polished as possible, we rolled along it a hard, smooth, and very round bronze ball. Having placed this board in a sloping position, by raising one end some one or two cubits above the other, we rolled the ball, as I was just saying, along the channel, noting, in a manner presently to be described, the time required to make the descent. We repeated this experiment more than once in order to measure the time with an accuracy such that the deviation between two observations never exceeded one-tenth of a pulse-beat. Having performed this operation and having assured ourselves of its reliability, we now rolled the ball only one-quarter the length of the channel; and having measured the time of its descent, we found it precisely one-half of the former. Next we tried other distances, compared the time for the whole length with that for the half, or with that for two-thirds, or three-fourths, or indeed for any fraction; in such experiments, repeated a full hundred times, we always found that the spaces traversed were to each other as the squares of the times, and this was true for all inclinations of the plane, i.e., of the channel, along which we rolled the ball. We also observed that the times of descent, for various inclinations of the plane, bore to one another precisely that ratio which, as we shall see later, the Author had predicted and demonstrated for them.

For the measurement of time, we employed a large vessel of water placed in an elevated position; to the bottom of this vessel was soldered a pipe of small diameter giving a thin jet of water which we collected in a small glass during the time of each descent, whether for the whole length of the channel or for part of its length; the water thus collected was weighed, after each descent, on a very accurate balance; the differences and ratios of these weights gave us the differences and ratios of the times, and this with such accuracy that although the operation was repeated many, many times, there was no appreciable discrepancy in the results.

So, Galileo's model was $$d=gt^2,$$ where $d$ is the distance traveled, $g$ - acceleration and $t$ - time. He would roll a ball at the full distance $d_0=1$ and establish the base time $t_0$. He proceeded to conduct 100 measurements at different $d_i$ measuring times $t_i$. Then he calculated $d_0/d_i$ and $t_0^2/t_i^2$. If his model was right then you'd have $$\frac{d_0}{d_i}=\frac{t_0^2}{t_i^2}$$.

Pay attention to how he measured time. It's so crude that it reminds me how these days unnatural sciences measure their variables, think of "customer satisfaction" or "utility". He mentions that the measurement error was within tenth of a unit of time, btw.

Did he include all relevant variables? Yes he did. Now, you have to understand that all bodies are attracted to each other by gravity. So, in theory to calculate the exact force on the ball you have to add every body in the universe to the equation. Moreover, much more importantly he didn't include surface resistance, air drag, angular momentum etc. Did these all impact his measurements? Yes. However, they were not relevant to what he was studying because he was able to reduce or eliminate their impact by isolating the impact of the property he was studying.

Now, Would you say that his coefficient (precisely 2 for $t^2$) was misleading because he "didn't control for air pressure and temperature changes between experiments"? No. Despite all the problems and limitations he was able to correctly establish the major law of motion, which still holds today at insane precision! He was able to accomplish this without statistical packages and computers because he designed a great experiment in such a way that the statistical part was rendered trivial and almost irrelevant. That's the idea situation you'd like to be.

Answer 3

For the assumptions of the regression model to hold perfectly, all relevant predictors must be included. But none of the assumptions in any statistical analysis hold perfectly and much of statistical practice is based on "Close Enough".

With Design of experiments and proper randomization, the effect of terms not included in the models can often be ignored (assumed equal by the chance of randomization). But, regression is usually used when full randomization is not possible to account for all possible variables not included in the model, so your question does become important.

Pretty much every regression model ever fit is probably missing some potential predictors, but "I don't Know" without any further clarification would not allow working statisticians to keep working, so we try our best and then try to work out how much the difference between the assumptions and reality will affect our results. In some cases the difference from the assumptions makes very little difference and we don't worry much about the difference, but in other cases it can be very serious.

One option when you know that there may be predictors that were not included in the model that would be relevant is to do a sensitivity analysis. This measures how much bias would be possible based on potential relationships with the unmeasured variable(s). This paper:

Lin, DY and Psaty, BM and Kronmal, RA. (1998): Assessing the Sensitivity of Regression Results to Unmeasured Confounders in Observational Studies. Biometrics, 54 (3), Sep, pp. 948-963.

gives some tools (and examples) of a sensitivity analysis.