# General expression for a single coefficient $\hat{\beta_1}$ in a multiple linear regression? [duplicate]

Suppose I am trying to estimate a multiple linear regression with $$k$$ regressors and I have $$n$$ observations

$$Y = X\beta + \epsilon$$

Where $$\beta \in \mathbb{R}^k$$ and $$X \in \mathbb{R}^{k \times n}$$.

The typical solution for the estimate of $$\beta$$ is written as $$\hat\beta = (X^TX)^{-1}X^TY$$. What I would like is a general expression for the estimator of $$\beta_1$$ given $$n$$ observations - an expression for $$\hat \beta_1$$, which is only one of the coefficients in $$\hat\beta$$.

A solution for two predictors has been given in this question. This questions ask for a formulation of the expression when there are more than two predictors.

I have split the above equation like

$$Y = \beta_0 + X_1\beta_1 + X_2\cdot\beta_2 + \epsilon$$

where $$\beta_0, \beta_1 \in \mathbb{R}$$, $$\beta_2 \in \mathbb{R}^{k-1}$$, $$X_1 \in \mathbb{R}^n$$ and $$X_2 \in \mathbb{R}^{(k-1) \times n}$$.

I have tried working with the matrix formulation, and also the method where I regress $$Y$$ on $$X_2$$, then $$X_1$$ on $$X_2$$ and then regress the residuals from the first regression on the residuals from the second regression, and then tried to work out an expression for $$\hat{\beta_1}$$ from that, but this is still incredibly messy.

I was just curious if there is a known expression for a single coefficient in a multiple regression (or if there is some easier way to derive it), which would save me a ton of time.

Use the standard results of partioned regression or equivalently FWL-theorem.

FWL-theorem

• The estimate of the coefficient $$\beta_1$$ in a regression model $$y = X_1 \beta_1 + X_2\beta_2 + e$$ will be the same as the estimate of $$\beta_1$$ in a regression model: $$\bar y = \bar X_1 \beta_1 + e$$ where $$\bar y$$ are the residuals of regressing $$y$$ on $$X_2$$ and $$\bar X_1$$ are the residuals of regressing $$X_1$$ on $$X_2$$.

To do so start by considering the regression model

$$y = x\beta + e = X_1 \beta_1 + X_2\beta_2 + e$$

where define design matrix $$X = [X_1 \ X_2]$$ and $$\beta = (\beta_1^\top \beta_2^\top)^\top$$. The normal equations defining the OLS estimatior is given as

$$X^\top y = X^\top X \hat \beta$$ and written in partioned form this becomes

$$\begin{bmatrix}X_1^\top y \\ X_2^\top y\end{bmatrix} = \begin{bmatrix}X_1^\top X_1 \hat \beta_1 + X_1^\top X_2\hat \beta_2 \\ X_2^\top X_1 \hat \beta_1 + X_2^\top X_2\hat \beta_2\end{bmatrix}$$

to find an expression for $$\hat \beta_1$$ use the lower part of this system of equations to get an expression for $$\hat \beta_2$$ which can be substituted into the upper part of the system. The lower part implies that

$$(X_2^\top X_2)^{-1}X_2^\top y - (X_2^\top X_2)^{-1}X_2^\top X_1 \hat \beta_1 = \hat \beta_2$$ and inserting in the upper part results in the expression

$$X_1^\top y = X_1^\top X_1 \hat \beta_1 + X_1^\top X_2[ (X_2^\top X_2)^{-1}X_2^\top y - (X_2^\top X_2)^{-1}X_2^\top X_1 \hat \beta_1 ]$$ defining the annihilator matrix (or residual maker matrix) $$M_{X_2} :=I- X_2 (X_2^\top X_2)^{-1}X_2^\top$$ this reduces to

$$X_1^\top M_{X_2} y = X_1^\top M_{X_2}X_1 \hat \beta_1$$

which can be solved to get an expression for $$\hat \beta_1$$

$$( X_1^\top M_{X_2}X_1 )^{-1}X_1^\top M_{X_2} y =\hat \beta_1 .$$

It is then noted that $$M_{X_2}$$ is idempotent and symmetric $$M_{X_2} = M_{X_2}M_{X_2} = M_{X_2}^\top$$ such that

$$\hat \beta_1 = ( X_1^\top M_{X_2}X_1 )^{-1}X_1^\top M_{X_2} y = ( X_1^\top M_{X_2}^\top M_{X_2}X_1 )^{-1}X_1^\top M_{X_2}^\top M_{X_2} y$$

and then defining $$\tilde X_1 := M_{X_2} X_1$$ and $$\tilde y := M_{X_2} y$$ it follows that

$$\hat \beta_1 = ( \tilde X_1^\top \tilde X_1 )^{-1}\tilde X_1^\top \tilde y$$

showing that $$\hat\beta_1$$ is the estimate you get if you regress $$\tilde y$$ on $$\tilde X_1$$. Furthermore the residual maker matrix $$M_{X_2}$$ is so called because it "computes" the residuals from a regression on $$X_2$$ so regressing $$y$$ on $$X_2$$ you get $$\hat \lambda= (X_2^\top X_2)^{-1} X_2^{\top}y$$ and predicted values $$X_2\hat \lambda = X_2(X_2^\top X_2)^{-1} X_2^{\top}y$$ and therefore resdiuals $$y - X_2(X_2^\top X_2)^{-1} X_2^{\top}y = M_{X_2}y$$. Similarly regressing the columns of $$X_1$$ on $$X_2$$ the same logic gives you residuals $$M_{X_2}X_1$$ which means that the estimate

$$\hat \beta_1 = ( \tilde X_1^\top \tilde X_1 )^{-1}\tilde X_1^\top \tilde y$$

is the estimate of regressing residuals from a regression of $$y$$ on $$X_2$$ on the residuals of regression of the columns of $$X_1$$ on $$X_2$$.

• You might want to place the short summary "which means that the estimate $\hat \beta_1 = ( \tilde X_1^\top \tilde X_1 )^{-1}\tilde X_1^\top \tilde y$ is the estimate of regressing residuals from a regression of $y$ on $X_2$ on the residuals of regression of the columns of $X_1$ on $X_2$." at the top of the question (start with the theorem instead with the proof). Dec 10 '19 at 11:00
• It is probably a good idea (you are welcome to edit as you see fit). But I fail to see the point in using time on improving on a post that has been - in my opinion wrongly - closed as a duplicate. Dec 10 '19 at 11:14
• I can see your standpoint about the closed case (this demotivates me as well when it happens). Anyway, I believe your question is providing a useful viewpoint (I gave a +1), therefore I believe it is useful to improve it (and an improvement that I could mention is that is is confusing to start with the proof before the theorem; I didn't know about these residuals and while reading your proof I was confused about the point of it). Dec 10 '19 at 12:09
• I have added the theorem at the top. You can remove it if you want. In addition, one could add a more intuitive explanation why this theorem works. The regressors $X_1$ extend the possible solution space and it is only the parts of $X_1$ and $Y$ that are orthogonal to $X_2$ that play a part in determining the coefficient $\beta_1$. We could write: $$\bar{X}^T_1 Y = \bar{X}^T_1 (X_1 \beta_1 + \underbrace{ X_2 \beta_2 + \epsilon}_{\substack{\text{orthogonal to \bar{X}_1 }\\\text{and will fall out }\\\text{ of the equation}}}) = \bar{X}^T_1 \bar X_1 \beta_1$$ Dec 10 '19 at 13:50