Exact equivalence of LR and Wald in linear regression under known error variance

Question

Is it true that the LR statistic and the Wald statistic are numerically equivalent when testing a nested hypothesis in a linear regression when the error variance is known? Hence, is a squared t-statistic equal to the corresponding LR statistic?

Answer 1

Yes.

[Remark: I took the liberty to ask and answer this question in order to be able to supply the missing parts in this answer. As I believe the present question could be of independent interest I thought it might be helpful to state it separately. The exposition is based on the textbook by Paul Ruud.]

Consider a partitioned linear regression model $$y=X_1\beta_{01}+X_2\beta_{02}+\epsilon,$$ where $y|X\sim\mathcal{N}(X\beta_0,\sigma^2I)$.

We test \begin{equation} H_0:\beta_{02}=0 \end{equation} The Wald test statistic is given by (see, e.g., here for the general formula) \begin{eqnarray*} \mathcal{W}&=&n\widehat{\beta}_2'\left[n\sigma^2\left[X_2'M_{X_1}X_2\right]^{-1}\right]^{-1}\widehat{\beta}_2\\ &=&\frac{\widehat{\beta}_2'X_2'M_{X_1}X_2\widehat{\beta}_2}{\sigma^2}\\ &=&\frac{y'M_{X_1}X_2(X_2'M_{X_1}X_2)^{-1}X_2'M_{X_1}X_2(X_2'M_{X_1}X_2)^{-1}X_2'M_{X_1}y}{\sigma^2}\\ &=&\frac{y'M_{X_1}X_2(X_2'M_{X_1}X_2)^{-1}X_2'M_{X_1}y}{\sigma^2}\\ &=&\frac{y'P_{M_{X_1}X_2}y}{\sigma^2}\\ &=:&\frac{y'P_{X_{2\bot1}}y}{\sigma^2}, \end{eqnarray*} where the third equality follows from the Frisch-Waugh-Lovell theorem. Here, $M_A$ and $P_A$ denote the usual residual maker and projection matrices on $A$.

We now give an expression for the Likelihood ratio test statistic under known error variances.

Inserting the restricted and unrestricted estimators, denoted $\widehat{\beta}$ and $\widehat{\beta}_R$, into the sample log-likelihood yields, using \begin{eqnarray*} L(\widehat{\beta})&=&-\frac{n}{2}\log\left(2\pi\sigma^2\right)-\frac{(y-X\widehat{\beta})'(y-X\widehat{\beta})}{2\sigma^2} \end{eqnarray*} and analogously for $L(\widehat{\beta}_R)$, the following expression for the $\mathcal{L}\mathcal{R}$-test statistic: \begin{eqnarray*}\mathcal{L}\mathcal{R}&=&2[L(\widehat{\theta})-L(\widehat{\theta}_R)]\\ &=&\frac{(y-X\widehat{\beta}_R)'(y-X\widehat{\beta}_R)-(y-X\widehat{\beta})'(y-X\widehat{\beta})}{\sigma^2}\\&=&\frac{y'(I-P_{X_1})y-y'(I-P_{X})y}{\sigma^2} \end{eqnarray*}

We now show that $\mathcal{W}$ may also be written in this format, \begin{eqnarray*} \mathcal{W}&=&\frac{y'(I-P_{X_1})y-y'(I-P_{X})y}{\sigma^2} \end{eqnarray*} We first show that $$ P_{X}=P_{X_1}+P_{X_{2\bot1}}, $$ as a partition of $X$, $$X=(X_{A}\vdots X_{B}),$$ in orthogonal matrices $X_{A}$, $X_{B}$ ($X_{A}'X_{B}=0$) satisfies that $$ P_{X}=P_{A}+P_{B} $$ This is so because \begin{eqnarray*} P_{X}&=&\left(\begin{array}{cc} X_{A} & X_{B} \\ \end{array} \right) \left( \begin{array}{cc} X_{A}'X_{A} & X_{A}'X_{B} \\ X_{B}'X_{A} & X_{B}'X_{B} \\ \end{array} \right)^{-1} \left( \begin{array}{c} X_{A}' \\ X_{B}' \\ \end{array} \right)\\ &=&\left(\begin{array}{cc} X_{A} & X_{B} \\ \end{array} \right) \left( \begin{array}{cc} X_{A}'X_{A} & 0 \\ 0 & X_{B}'X_{B} \\ \end{array} \right)^{-1} \left( \begin{array}{c} X_{A}' \\ X_{B}' \\ \end{array} \right)\\ &=&\left(\begin{array}{cc} X_{A} & X_{B} \\ \end{array} \right) \left( \begin{array}{cc} (X_{A}'X_{A})^{-1} & 0 \\ 0 & (X_{B}'X_{B})^{-1} \\ \end{array} \right) \left( \begin{array}{c} X_{A}' \\ X_{B}' \\ \end{array} \right)\\ &=& X_{A}(X_{A}'X_{A})^{-1}X_{A}'+X_{B}(X_{B}'X_{B})^{-1}X_{B}'\\ &=&P_{A}+P_{B} \end{eqnarray*} We can apply this intermediate result to $X_1$ and $X_{2\bot1}$, as $X_{2\bot1}'X_1=0$. Hence, $$P_{X_{2\bot1}}=P_{X}-P_{X_1}.$$ Adding and subtracting $y'Iy$ in the numerator of the Wald statistic completes the proof.

It may be seen from, e.g., this answer that $\mathcal{W}=t^2$, with $t$ the t-ratio for some single coefficient, when we replace $s^2$ by $\sigma^2$, assumed known. When we need to estimate $\sigma^2$, it is however no longer true that $\mathcal{W}=\mathcal{L}\mathcal{R}$.

It may be established that the score test statistic is also numerically equivalent, which is skipped here for brevity.