Interpretation of moderately correlated predictors in linear model

I have understood why it's a bad idea having highly correlated predictors, what is puzzling me is a meaningful interpretation of moderately correlated predictors ($r< 0.3$). Let's suppose we have fitted the following linear model to some data after highly correlated predictors have been removed: $$y = c_1x_1 + c_2x_2 + c_3x_3 + c_4x_4 + c_5x_5$$ (For simplicity the intercept is assumed to be $0$.)

My understanding is that a way of reporting the above model would be something like:

unit change in predictor $x_3$ will results in a $c_3$ change in $y$.

However it seems to me that if predictors are moderately correlated and there are enough of them a unit change in $x_3$ in reality will correspond to a change in the other predictors and so the effect on $y$ could end up being completely different.This simply because a pure unit change in $x_3$ cannot happen without the other predictors changing due to correlation. Because of this fact I cannot see how this interpretation can be meaningful when applied to reality.

Could anybody clarify how to meaningfully report such models? I have misunderstood something perhaps?

The interpretation you list is correct. It is still applicable with correlated variables.

This is easier for me to show with an example. Suppose you have income and years of education as independent variables. It is true that these are correlated. However, it is possible to be at any combination. There are high school dropouts who are millionaires and PhDs who make minimum wage and everything in between.

This would only be strictly impossible if the correlation were 1 or -1, but it gets harder as correlation moves away from 0.

Diagnosing collinearity is a notoriously slippery topic. Since results are conditional on the other factors in the model, the magnitude of pairwise correlations are not reliable as a diagnostic metric for collinearity in regression -- don't use them. Partial or even semi-partial correlations are much more reliable indicators. Of course, VIFs and indexed eigenvalues as recommended by Belsey, Kuh and Welsch in their classic book Regression Diagnostics are probably the most clinical tools but even they are fallible.

• Belsley doesn't recommend VIFs, he recommends condition indexes and proportion of variance explained (I assume this is the same as indexed eigenvalues, but I don't recall hearing them called that). In what way are they fallible? Do you like the perturb package? – Peter Flom Apr 4 '16 at 12:24
• @PeterFlom I'll have to take your word that BKW don't reference VIFs as I haven't read their book in quite a while. As for condition indexes, the lack of agreement wrt the appropriate threshold(s) to use in deciding whether or not collinearity is present is a major fallibility. They provide rules of thumb for this. Finally, I don't know the perturb package. Will have to look it up. Thanks! – Mike Hunter Apr 4 '16 at 12:27

Your question raises a number of interesting questions about the use (or abuse) of linear regression in observational studies. Regression analysis assumes that the independent variables are fixed, not stochastic. In a designed experiment, the values are set by the researcher. Mathematically, correlations can be calculated between these variables, but there is no random process going on. It would be better to describe these variables as non-orthogonal, rather than "correlated".

And yes, orthogonality matters, because it allows you to make inferences about each independent variable without having to reference the other variables. So to answer your question, when there is non-orthogonality, the interpretation of, say, $c_1$ in your model is that it measures the impact of $x_1$ in a model that already contains the other variables. And $c_2$ is the impact of $x_2$ in a model that already contains $x_1, x_3, x_4, x_5$. This is not always the answer to a research question of interest. And yes, mathematically, each coefficient conveys the result on $y$ of a unit change in an $x$ when all other variables are held fixed.

In an observational study, we model $y, x_1, x_2, x_3, x_4, x_5$ as a multivariate observation, and the regression model is basically an attempt to describe the conditional distribution of $y$. You make a valid point about how, in practice, if $x_1$ changes, the other $x$ variables will typically change as well. But that's not what the conditional distribution is trying to capture. In fact, if you want to know how $y$ varies, taking into account the real world variation in the $x$ variables, you only need to look at the marginal distribution of $y$ - no regressions needed! The distribution of $y$ contains, implicitly, the collective impact of the $x$ variables, as they occur in the real world, with all their mutual correlations.

Personally, I have a lot of reservations about using linear regression on observational data. There are historical reasons why we do this, but I am relieved to see the growing adoption of latent variable and measurement error models, which do not treat independent variables as if they had been fixed in advance, without error.

• @placida Given your comments, you would probably enjoy Richard Nisbett's Crusade Against Multiple Regression... edge.org/conversation/… – Mike Hunter Apr 4 '16 at 17:38