For simplicity, let's restrict multiple linear regression case to 2 predictors, $x_1, x_2$. You regress $y$ on each individually and get $\hat{\beta}_1, \hat{\beta}_2$. Now you regress $y$ on both and get $\hat{\gamma}_1, \hat{\gamma}_2$.

So I know if $(x_1 - \bar{x}_1 \perp \bar{x}_2$, then $\hat{\beta}_i = \hat{\gamma}_i$, but if they're not orthogonal, what can be said about the relationship between them?

If in each of the simple linear regression cases, the slope was positive, i.e., $\hat{\beta}_1, \hat{\beta_2} > 0$, can we expect $\hat{\gamma}_1, \hat{\gamma}_2 > 0$?

I just asked this question on the math SE (https://math.stackexchange.com/questions/3791992/relationship-between-projection-of-y-onto-x-1-x-2-individually-vs-projecti), but I'm looking for more of a linear algebra intuition in that question. Here, I'm opening for any kind of intuition, statistical or not.

  • $\begingroup$ See Frisch–Waugh–Lovell theorem and Simpsons paradox. $\endgroup$ Aug 15 '20 at 17:40

Here is a simple example that provides insight.

y = c(5.8,5.2,4.7,8.7,8.1,7.7,10.2,9.6,9.0)
x1 = c(1,1.5,2,1.8,2.7,3.5,3,4,4.5)
x2 = c(1,1,1,2,2,2,3,3,3)


legend("topleft", c("x2=1", "x2=2", "x2=3"), pch=1, col=1:3)

The simple regressions have significant positive relationships, but the multiple regression shows that the effect of x1 is significant and negative. The graph gives the intuition clearly:

Multiple Regression Graph

Ignoring x1, there are generally higher values of y for larger x2. Similarly, ignoring x2, there are generally larger values of y for larger x1. These observations explain the simple regression results.

In the multiple regression model, the slope coefficients are estimates of the effect of one x while the other is held fixed. And you can easily see in the graph that the values of y are smaller as x1 increases within any of the three groups where x2 is held fixed (at either 1,2, or 3).

  • $\begingroup$ "In the multiple regression model, the slope coefficients are estimates of the effect of one x while the other is held fixed" In the simple linear regression model, isn't the unmodeled $x$ technically fixed too? I don't think I understand what "fixed" entails in this context. $\endgroup$
    – roulette01
    Aug 15 '20 at 18:42
  • $\begingroup$ @dd2205 “fixed” meaning what? I take “fixed” to mean in the sense that we hold other variables fixed when we take partial derivatives in multivariable calculus. $\endgroup$
    – Dave
    Aug 15 '20 at 19:02
  • 1
    $\begingroup$ The unmodeled x is definitely not fixed in the simple models. That's why the slope coefficients are both positive in my example. The reason for the "fixed" concept lies in the model itself: $E(y|x_1,x_2) = \beta_0 + \beta_1 x_1 + \beta_2x_2$ implies that, eg, $E(y|x_1,x_2=1) = \beta_0 + \beta_1 x_1 + \beta_2 (1) = (\beta_0 + \beta_2) + \beta_1 x_1$. So $\beta_1$ in the multiple regression model is the effect of $x_1$ when $x_2$ is fixed (at any value, not just 1.) $\endgroup$ Aug 15 '20 at 19:05
  • $\begingroup$ Yes, you can think of the "held fixed" concept in terms of partial derivatives as well. $\endgroup$ Aug 15 '20 at 19:07
  • $\begingroup$ This generalizes to any number of predictors right? For example, if we had $p$ predictors, we can interpret $\beta_1$ as the effect of $x_1$ when $x_2, \ldots, x_p$ are fixed? $\endgroup$
    – roulette01
    Aug 15 '20 at 19:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.