# Restricted OLS problem

Here is my problem:

Let $1\leq l<k$ be integers and write the design matrix $X$ as $X=[X_1\; X_2]$, where $X_1$ is an $n\times l$-dimensional matrix, and where $X_2$ is an $n\times (k-l)$-dimensional matrix. Show that the restricted LS estimator $\beta^*$ subject to the restriction $R\beta=r$ with $R=[I_l\;\, 0]$ and $r=0$ satisfies $\beta^*=(0,[(X_2'X_2)^{-1}X_2'Y]')'$.

My solution: $$(X^TX)=\left( \begin{array} {c,c} X_1^TX_1 \quad X_1^TX_2 \\ X_2^TX_1 \quad X_2^TX_2 \end{array} \right)$$

$$R(X^TX)^{-1}R^T = (X_1^TX_1 - X_1^TX_2(X_2^TX_2)^{-1}X_2^TX_1)^{-1}$$

$$R\hat\beta - r = (I_l:0)\hat\beta = (\hat\beta_l :0)$$

$$(X^TX)^{-1}R^T=\left( \begin{array} {c,c} X_1^TX_1(X_1^TX_1 - X_1^TX_2(X_2^TX_2)^{-1}X_2^TX_1)^{-1} \\ (X_2^TX_2)^{-1}X_2X_1((X_1^TX_1 - X_1^TX_2(X_2^TX_2)^{-1}X_2^TX_1)^{-1} \quad \end{array} \right)$$

But then when I multiply all terms I got

$$\beta^*= \hat\beta - \left( \begin{array} {c,c} \hat\beta \quad 0 \\ BA\hat\beta_ \quad 0 \end{array} \right)$$

where $B = (X_2^TX_2)^{-1}X_2X_1((X_1^TX_1 - X_1^TX_2(X_2^TX_2)^{-1}X_2^TX_1)^{-1}$

and $A = (X_1^TX_1 - X_1^TX_2(X_2^TX_2)^{-1}X_2^TX_1)^{-1}$

So I during multiplication I did something wrong. Can you give me a hint, how should I solve this problem(maybe some clever trick in order to make multiplication a bit easier)?

## 1 Answer

The restriction just says that the variables in $X_1$ do not enter the regression, so that the formula constrains those coefficients to zero, and the others then are the standard OLS coefficients of a regression of $y$ on $X_2$.

Here is an implementation in R (a more formal answer follows below):

library(lrmest)
# example data
n <- 100
k_1 <- 3
k_2 <- 4
X_1 <- matrix(rnorm(n*k_1),ncol=k_1)
X_2 <- matrix(rnorm(n*k_2),ncol=k_2)
y <- rnorm(n)
X <- cbind(X_1,X_2)

# implementing the restriction in lm directly
coef(lm(y~X_2-1))

>
X_21        X_22        X_23        X_24
-0.02171519 -0.04595167  0.08971665  0.04871940

# restricted least squares
R <- cbind(diag(k_1),matrix(0,nrow=k_1,ncol=k_2))
r <- delta <- rep(0,k_1)
rls(y~X-1,r,R,delt=delta)

>
$*****Restricted Least Square Estimator***** Estimate Standard_error t_statistic pvalue X1 0.0000 0.0000 0 1 X2 0.0000 0.0000 0 1 X3 0.0000 0.0000 0 1 X4 -0.0217 0.0960 NA NA X5 -0.0460 0.1091 NA NA X6 0.0897 0.1093 NA NA X7 0.0487 0.0922 NA NA  Both question and answer make use of properties of restricted OLS, which are for example discussed further here. In particular, the restricted OLS estimator is related to the unrestricted one via $$\beta^*=\hat\beta+(X^TX)^{-1}R^T[R(X^TX)^{-1}R^T]^{-1}(r-R\hat\beta)$$ Algebraically, the mistake seems to be in $$R\hat\beta - r = (I_l:0)\hat\beta,$$ which is just$\hat\beta_l. In more detail, denote \begin{align*} (X^TX)^{-1}&=\left( \begin{array} {c,c} X_1^TX_1&X_1^TX_2 \\ X_2^TX_1&X_2^TX_2\end{array} \right)^{-1}\\&\equiv\left( \begin{array} {c,c} \tilde A&\tilde B \\ \tilde C&\tilde D\end{array} \right) \end{align*} Then, $$R(X^TX)^{-1}R^T = \tilde A$$ and $$(X^TX)^{-1}R^T = \left( \begin{array} {c} \tilde A \\ \tilde C\end{array} \right)$$ leading to \begin{align*} \beta^*&= \left( \begin{array} {c} \hat\beta_l \\ \hat\beta_k\end{array} \right) - \left( \begin{array} {c,c} \tilde A \\ \tilde C\end{array} \right)\tilde A^{-1}R\hat\beta\\ &= \left( \begin{array} {c} \hat\beta_l \\ \hat\beta_k\end{array} \right) - \left( \begin{array} {c,c} I \\ \tilde C\tilde A^{-1}\end{array} \right)\hat\beta_l\\ \end{align*} Thus, using $$\hat\beta=\left( \begin{array} {c,c} \tilde A&\tilde B \\ \tilde C&\tilde D\end{array} \right)\left( \begin{array} {c} X_1^Ty\\X_2^Ty\end{array} \right),$$ we have $$\hat\beta_k=\tilde C X_1^Ty+\tilde DX_2^Ty$$ and $$\hat\beta_l=\tilde A X_1^Ty+\tilde BX_2^Ty$$ so that $$\tilde C\tilde A^{-1}\hat\beta_l=\tilde C\tilde A^{-1}\tilde A X_1^Ty+\tilde C\tilde A^{-1}\tilde BX_2^Ty$$ and \begin{align*} \hat\beta_k-\tilde C\tilde A^{-1}\hat\beta_l&=\tilde C X_1^Ty+\tilde DX_2^Ty- \tilde C\tilde A^{-1}\tilde A X_1^Ty-\tilde C\tilde A^{-1}\tilde BX_2^Ty\\ &=(\tilde D-\tilde C\tilde A^{-1}\tilde B)X_2^Ty \end{align*} Now, use results for partitioned inverses \begin{align*} (X^TX)^{-1} &=\left( \begin{array} {c,c} \tilde A&\tilde B \\ \tilde C&\tilde D\end{array} \right)\\&= \left( \begin{array} {c,c} A&B \\ C&D\end{array} \right)^{-1}\\ &=\left( \begin{array} {c,c} E&-EBD^{-1} \\ -D^{-1}CE&D^{-1}+D^{-1}CEBD^{-1}\end{array} \right)\\ \end{align*} whereE=(A-BD^{-1}C)^{-1}\$.

Hence, \begin{align*} \tilde D-\tilde C\tilde A^{-1}\tilde B&=D^{-1}+D^{-1}CEBD^{-1}-D^{-1}CEE^{-1}EBD^{-1}\\&=D^{-1} \end{align*} The result is complete by noting that $$D^{-1}=(X_2^TX_2)^{-1}$$ so that $$\hat\beta_k-\tilde C\tilde A^{-1}\hat\beta_l=(X_2^TX_2)^{-1}X_2^Ty$$

• Thank you very much for the full description of problem:) Very helpful! Commented Jan 5, 2017 at 11:14