Let's set up the situation of having some $Y$ that I think depends on a linear combination of $X_1$ and $X_2$. I could fit a regression model:

$$y_i = \beta_0 + \beta_1x_{i1} + \beta_2x_{i2}$$

We could write this as a function of the predictor variables:

$$y(x_1, x_2) = \beta_0 + \beta_1x_{1} + \beta_2x_{2}$$

Then we would interpret the coefficients as being the partial derivatives.

$$\dfrac{\partial y}{\partial x_1} = \beta_1$$

$$\dfrac{\partial y}{\partial x_2} = \beta_2$$

This is consistent with our usual idea that, as we increase $x_1$ by one unit and leave $x_2$ alone, $y$ changes by $\beta_1$.

However, what if $X_1$ and $X_2$ are correlated? In that case, if we increase $x_1$ by one unit, $x_2$ should change by some amount.

I've gotten as far as thinking that it has something to do with the inner product of $\big(\beta_1, \beta_2\big)$ with itself with respect to the covariance matrix of $X_1$ and $X_2$:

$$\begin{pmatrix} \beta_1 & \beta_2 \end{pmatrix} \begin{pmatrix} \sigma_{1,1} & \sigma_{1,2}\\ \sigma_{2,1} & \sigma_{2,2} \end{pmatrix} \begin{pmatrix} \beta_1 \\ \beta_2 \end{pmatrix} $$

Thoughts? How does the intercept play into this? Certainly the intercept should drop out, but where?

(There should be $\widehat{\text{hats}}$ all over the place, yes.)

  • $\begingroup$ Yes, please do add the hats in the $y$ and $\beta$s $\endgroup$ – Tim Mak Apr 8 '20 at 2:53

The coefficients in a multiple linear regression are by definition conditional coefficients. If you want the marginal relationship, the general answer is to integrate over the distribution of $x_1$ and $x_2$. However, typically, the distribution is unspecified, and people use the empirical distribution instead. In the linear regression case, I think this reduces to simply fitting the model of one variable without the other.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.