3
$\begingroup$

Suppose I am trying to estimate a multiple linear regression with $k$ regressors and I have $n$ observations

$$Y = X\beta + \epsilon$$

Where $\beta \in \mathbb{R}^k$ and $X \in \mathbb{R}^{k \times n}$.

The typical solution for the estimate of $\beta$ is written as $\hat\beta = (X^TX)^{-1}X^TY$. What I would like is a general expression for the estimator of $\beta_1$ given $n$ observations - an expression for $\hat \beta_1$, which is only one of the coefficients in $\hat\beta$.

A solution for two predictors has been given in this question. This questions ask for a formulation of the expression when there are more than two predictors.


I have split the above equation like

$$Y = \beta_0 + X_1\beta_1 + X_2\cdot\beta_2 + \epsilon$$

where $\beta_0, \beta_1 \in \mathbb{R}$, $\beta_2 \in \mathbb{R}^{k-1}$, $X_1 \in \mathbb{R}^n$ and $X_2 \in \mathbb{R}^{(k-1) \times n}$.

I have tried working with the matrix formulation, and also the method where I regress $Y$ on $X_2$, then $X_1$ on $X_2$ and then regress the residuals from the first regression on the residuals from the second regression, and then tried to work out an expression for $\hat{\beta_1}$ from that, but this is still incredibly messy.

I was just curious if there is a known expression for a single coefficient in a multiple regression (or if there is some easier way to derive it), which would save me a ton of time.

$\endgroup$
3
$\begingroup$

Use the standard results of partioned regression or equivalently FWL-theorem.


FWL-theorem

  • The estimate of the coefficient $\beta_1$ in a regression model $$y = X_1 \beta_1 + X_2\beta_2 + e$$ will be the same as the estimate of $\beta_1$ in a regression model: $$\bar y = \bar X_1 \beta_1 + e$$ where $\bar y$ are the residuals of regressing $y$ on $X_2$ and $\bar X_1$ are the residuals of regressing $X_1$ on $X_2$.

To do so start by considering the regression model

$$y = x\beta + e = X_1 \beta_1 + X_2\beta_2 + e$$

where define design matrix $X = [X_1 \ X_2]$ and $\beta = (\beta_1^\top \beta_2^\top)^\top$. The normal equations defining the OLS estimatior is given as

$$X^\top y = X^\top X \hat \beta $$ and written in partioned form this becomes

$$ \begin{bmatrix}X_1^\top y \\ X_2^\top y\end{bmatrix} = \begin{bmatrix}X_1^\top X_1 \hat \beta_1 + X_1^\top X_2\hat \beta_2 \\ X_2^\top X_1 \hat \beta_1 + X_2^\top X_2\hat \beta_2\end{bmatrix}$$

to find an expression for $\hat \beta_1$ use the lower part of this system of equations to get an expression for $\hat \beta_2$ which can be substituted into the upper part of the system. The lower part implies that

$$ (X_2^\top X_2)^{-1}X_2^\top y - (X_2^\top X_2)^{-1}X_2^\top X_1 \hat \beta_1 = \hat \beta_2$$ and inserting in the upper part results in the expression

$$ X_1^\top y = X_1^\top X_1 \hat \beta_1 + X_1^\top X_2[ (X_2^\top X_2)^{-1}X_2^\top y - (X_2^\top X_2)^{-1}X_2^\top X_1 \hat \beta_1 ]$$ defining the annihilator matrix (or residual maker matrix) $M_{X_2} :=I- X_2 (X_2^\top X_2)^{-1}X_2^\top$ this reduces to

$$ X_1^\top M_{X_2} y = X_1^\top M_{X_2}X_1 \hat \beta_1 $$

which can be solved to get an expression for $\hat \beta_1$

$$ ( X_1^\top M_{X_2}X_1 )^{-1}X_1^\top M_{X_2} y =\hat \beta_1 .$$

It is then noted that $M_{X_2}$ is idempotent and symmetric $M_{X_2} = M_{X_2}M_{X_2} = M_{X_2}^\top$ such that

$$ \hat \beta_1 = ( X_1^\top M_{X_2}X_1 )^{-1}X_1^\top M_{X_2} y = ( X_1^\top M_{X_2}^\top M_{X_2}X_1 )^{-1}X_1^\top M_{X_2}^\top M_{X_2} y$$

and then defining $\tilde X_1 := M_{X_2} X_1$ and $\tilde y := M_{X_2} y$ it follows that

$$ \hat \beta_1 = ( \tilde X_1^\top \tilde X_1 )^{-1}\tilde X_1^\top \tilde y$$

showing that $\hat\beta_1$ is the estimate you get if you regress $\tilde y$ on $\tilde X_1$. Furthermore the residual maker matrix $M_{X_2}$ is so called because it "computes" the residuals from a regression on $X_2$ so regressing $y$ on $X_2$ you get $\hat \lambda= (X_2^\top X_2)^{-1} X_2^{\top}y$ and predicted values $X_2\hat \lambda = X_2(X_2^\top X_2)^{-1} X_2^{\top}y$ and therefore resdiuals $y - X_2(X_2^\top X_2)^{-1} X_2^{\top}y = M_{X_2}y$. Similarly regressing the columns of $X_1$ on $X_2$ the same logic gives you residuals $M_{X_2}X_1$ which means that the estimate

$$ \hat \beta_1 = ( \tilde X_1^\top \tilde X_1 )^{-1}\tilde X_1^\top \tilde y$$

is the estimate of regressing residuals from a regression of $y$ on $X_2$ on the residuals of regression of the columns of $X_1$ on $X_2$.

$\endgroup$
  • 1
    $\begingroup$ You might want to place the short summary "which means that the estimate $\hat \beta_1 = ( \tilde X_1^\top \tilde X_1 )^{-1}\tilde X_1^\top \tilde y$ is the estimate of regressing residuals from a regression of $y$ on $X_2$ on the residuals of regression of the columns of $X_1$ on $X_2$." at the top of the question (start with the theorem instead with the proof). $\endgroup$ – Sextus Empiricus Dec 10 '19 at 11:00
  • $\begingroup$ It is probably a good idea (you are welcome to edit as you see fit). But I fail to see the point in using time on improving on a post that has been - in my opinion wrongly - closed as a duplicate. $\endgroup$ – Stop Closing Questions Fast Dec 10 '19 at 11:14
  • $\begingroup$ I can see your standpoint about the closed case (this demotivates me as well when it happens). Anyway, I believe your question is providing a useful viewpoint (I gave a +1), therefore I believe it is useful to improve it (and an improvement that I could mention is that is is confusing to start with the proof before the theorem; I didn't know about these residuals and while reading your proof I was confused about the point of it). $\endgroup$ – Sextus Empiricus Dec 10 '19 at 12:09
  • $\begingroup$ I have added the theorem at the top. You can remove it if you want. In addition, one could add a more intuitive explanation why this theorem works. The regressors $X_1$ extend the possible solution space and it is only the parts of $X_1$ and $Y$ that are orthogonal to $X_2$ that play a part in determining the coefficient $\beta_1$. We could write: $$\bar{X}^T_1 Y = \bar{X}^T_1 (X_1 \beta_1 + \underbrace{ X_2 \beta_2 + \epsilon}_{\substack{\text{orthogonal to $\bar{X}_1$ }\\\text{and will fall out }\\\text{ of the equation}}}) = \bar{X}^T_1 \bar X_1 \beta_1$$ $\endgroup$ – Sextus Empiricus Dec 10 '19 at 13:50

Not the answer you're looking for? Browse other questions tagged or ask your own question.