Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

A basic assumption of using regression models for inference is that "all relevant predictors" have been included in the prediction equation. The rationale is that failure to include an important real-world factor leads to biased coefficients and thus inaccurate inferences (i.e, omitted variable bias).

But in research practice, I have never seen anyone including anything resembling "all relevant predictors." Many phenomena have a myriad of important causes, and it would be very difficult, if not impossible, to include them all. An off-the-cuff example is modeling depression as an outcome: No one has built anything close to a model that includes "all relevant variables": e.g., parental history, personality traits, social support, income, their interactions, etc., etc...

Moreover, fitting such a complex model would lead to highly unstable estimates unless there were very large sample sizes.

My question is very simple: Is the assumption/advice to "include all relevant predictors" just something that we "say" but never actually mean? If not, then why do we give it as actual modeling advice?

And does this mean that most coefficients are probably misleading? (e.g., a study on personality factors and depression that uses only several predictors). In other words, how big of a problem is this for the conclusions of our sciences?

share|improve this question
4  
A version of this argument raged in psychology, economics, and the social sciences throughout the 19th century. Scientists argued that statistical methods were inapplicable to people and social systems because people are too varied and complex. By the end of that century, utility won out over philosophy: even when we don't include all predictors, we still can learn a lot. There is wisdom in the inclusion of "relevant" in the title phrase. – whuber Mar 28 at 20:46
up vote 13 down vote accepted

You are right - we are seldom realistic in saying "all relevant predictors". In practice we can be satisfied with including predictors that explain the major sources of variation in $Y$. In the special case of drawing inference about a risk factor or treatment in an observational study, this is seldom good enough. For that, adjustment for confounding needs to be highly agressive, including variables that might be related to outcome and might be related to treatment choice or to the risk factor you are trying to publicize.

It is interested that with the normal linear model, omitted covariates, especially if orthogonal to included covariates, can be thought of as just enlarging the error term. In nonlinear models (logistic, Cox, many others) omission of variables can bias the effects of all the variables included in the model (due to non-collapsibility of the odds ratio, for example).

share|improve this answer
    
Thank you for the helpful information. Setting aside assessing treatment effects, I'd like to ask more about the pragmatic implications of this issue. If you reviewed a paper and there were clearly important predictors omitted, could that be grounds for rejection? I ask this because a.) I've never heard of this occurring and b.) social scientists often include ONLY the predictors that they wish to know more about (i.e., the topic of the study) and neglect "already known" factors (based on the need for more efficient measurement). – ATJ Mar 28 at 21:45
    
For instance, it's not uncommon to see a latent variable model with only a SINGLE predictor for an endogenous variable. Does this speak to the gap between the field of statistics and its implementation in actual topic areas? – ATJ Mar 28 at 21:53
5  
It probably does. To the earlier question grounds for rejection would include omission of important variables whose inclusion would have given a different interpretation of included variables, or which would have drastically change the model. I once reviewed a paper on lung cancer risk that only had available whether or not a subject had ever smoked, and the authors did not attempt to assess the dose of smoking (e.g., pack-years). I recommended outright rejection. – Frank Harrell Mar 29 at 2:50

Yes, you must include all "relevant variables", but you must be smart about it. You must think of the ways to construct the experiments that would isolate the impact of your phenomenon from unrelated stuff, which is a plenty in real world (as opposed to a class room) research. Before you get into statistics, you have to do the heavy lifting in your domain, not in statistics.

I encourage you not be cynical about including all relevant variables, because it's not only a noble goal but also because it's often possible. We don't say this just for the sake of saying it. We really do mean it. In fact, designing experiments and studies that are able to include all relevant variables is what makes science really interesting, and different from mechanical boiler plate "experiments".

To motivate my statement, I'll give you an example of how Galileo studied acceleration. Here's his description of an actual experiment (from this web page):

A piece of wooden moulding or scantling, about 12 cubits long, half a cubit wide, and three finger-breadths thick, was taken; on its edge was cut a channel a little more than one finger in breadth; having made this groove very straight, smooth, and polished, and having lined it with parchment, also as smooth and polished as possible, we rolled along it a hard, smooth, and very round bronze ball. Having placed this board in a sloping position, by raising one end some one or two cubits above the other, we rolled the ball, as I was just saying, along the channel, noting, in a manner presently to be described, the time required to make the descent. We repeated this experiment more than once in order to measure the time with an accuracy such that the deviation between two observations never exceeded one-tenth of a pulse-beat. Having performed this operation and having assured ourselves of its reliability, we now rolled the ball only one-quarter the length of the channel; and having measured the time of its descent, we found it precisely one-half of the former. Next we tried other distances, compared the time for the whole length with that for the half, or with that for two-thirds, or three-fourths, or indeed for any fraction; in such experiments, repeated a full hundred times, we always found that the spaces traversed were to each other as the squares of the times, and this was true for all inclinations of the plane, i.e., of the channel, along which we rolled the ball. We also observed that the times of descent, for various inclinations of the plane, bore to one another precisely that ratio which, as we shall see later, the Author had predicted and demonstrated for them.

For the measurement of time, we employed a large vessel of water placed in an elevated position; to the bottom of this vessel was soldered a pipe of small diameter giving a thin jet of water which we collected in a small glass during the time of each descent, whether for the whole length of the channel or for part of its length; the water thus collected was weighed, after each descent, on a very accurate balance; the differences and ratios of these weights gave us the differences and ratios of the times, and this with such accuracy that although the operation was repeated many, many times, there was no appreciable discrepancy in the results.

So, Galileo's model was $$d=gt^2,$$ where $d$ is the distance traveled, $g$ - acceleration and $t$ - time. He would roll a ball at the full distance $d_0=1$ and establish the base time $t_0$. He proceeded to conduct 100 measurements at different $d_i$ measuring times $t_i$. Then he calculated $d_0/d_i$ and $t_0^2/t_i^2$. If his model was right then you'd have $$\frac{d_0}{d_i}=\frac{t_0^2}{t_i^2}$$.

Pay attention to how he measured time. It's so crude that it reminds me how these days unnatural sciences measure their variables, think of "customer satisfaction" or "utility". He mentions that the measurement error was within tenth of a unit of time, btw.

Did he include all relevant variables? Yes he did. Now, you have to understand that all bodies are attracted to each other by gravity. So, in theory to calculate the exact force on the ball you have to add every body in the universe to the equation. Moreover, much more importantly he didn't include surface resistance, air drag, angular momentum etc. Did these all impact his measurements? Yes. However, they were not relevant to what he was studying because he was able to reduce or eliminate their impact by isolating the impact of the property he was studying.

Now, Would you say that his coefficient (precisely 2 for $t^2$) was misleading because he "didn't control for air pressure and temperature changes between experiments"? No. Despite all the problems and limitations he was able to correctly establish the major law of motion, which still holds today at insane precision! He was able to accomplish this without statistical packages and computers because he designed a great experiment in such a way that the statistical part was rendered trivial and almost irrelevant. That's the idea situation you'd like to be.

share|improve this answer
    
What's so crude about his method of time measure? The setup will have a specific rate at which water will leave the large vessel and enter the cup; assuming the vessel contains a large quantity of water, that rate will change minimally. More importantly, it will remain consistent across experiments. It's actually a very elegant method given they didn't have stopwatches and fancy automatic timers back then. – JAB Mar 29 at 14:48
    
@JAB, it's crude only in comparison to stop watch or modern ways to measure time, of course. You're absolutely right that it's very elegant given the state of the art of time measurement in Galileo's days. However, the point I was making was that even such seemingly low precision (1/10 of an interval) was still enough to observe the relation between time and distance – Aksakal Mar 29 at 15:44
    
@JAB, one of my favorite examples of ridiculous measurement methods in physics is in how Cherenkov discovered his radiation. He'd sit in a dark room until his eyes are adjusted to darkness, then he'd opening or closing the hole with light coming from it until the light disappears. He'd be recording how much of a hole was open to detect the level of radiation. Apparently, human eye can detect the difference in light measured in a handful of photons! The paper is 3 pages long. – Aksakal Mar 29 at 16:01

For the assumptions of the regression model to hold perfectly, all relevant predictors must be included. But none of the assumptions in any statistical analysis hold perfectly and much of statistical practice is based on "Close Enough".

With Design of experiments and proper randomization, the effect of terms not included in the models can often be ignored (assumed equal by the chance of randomization). But, regression is usually used when full randomization is not possible to account for all possible variables not included in the model, so your question does become important.

Pretty much every regression model ever fit is probably missing some potential predictors, but "I don't Know" without any further clarification would not allow working statisticians to keep working, so we try our best and then try to work out how much the difference between the assumptions and reality will affect our results. In some cases the difference from the assumptions makes very little difference and we don't worry much about the difference, but in other cases it can be very serious.

One option when you know that there may be predictors that were not included in the model that would be relevant is to do a sensitivity analysis. This measures how much bias would be possible based on potential relationships with the unmeasured variable(s). This paper:

Lin, DY and Psaty, BM and Kronmal, RA. (1998): Assessing the Sensitivity of Regression Results to Unmeasured Confounders in Observational Studies. Biometrics, 54 (3), Sep, pp. 948-963.

gives some tools (and examples) of a sensitivity analysis.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.