3
$\begingroup$

Is it true that the LR statistic and the Wald statistic are numerically equivalent when testing a nested hypothesis in a linear regression when the error variance is known? Hence, is a squared t-statistic equal to the corresponding LR statistic?

$\endgroup$
1
$\begingroup$

Yes.

[Remark: I took the liberty to ask and answer this question in order to be able to supply the missing parts in this answer. As I believe the present question could be of independent interest I thought it might be helpful to state it separately.]

Consider a partitioned linear regression model $$y=X_1\beta_{01}+X_2\beta_{02}+\epsilon,$$ where $y|X\sim\mathcal{N}(X\beta_0,\sigma^2I)$.

We test \begin{equation} H_0:\beta_{02}=0 \end{equation} The Wald test statistic is given by (see, e.g., here for the general formula) \begin{eqnarray*} \mathcal{W}&=&n\widehat{\beta}_2'\left[n\sigma^2\left[X_2'M_{X_1}X_2\right]^{-1}\right]^{-1}\widehat{\beta}_2\\ &=&\frac{\widehat{\beta}_2'X_2'M_{X_1}X_2\widehat{\beta}_2}{\sigma^2}\\ &=&\frac{y'M_{X_1}X_2(X_2'M_{X_1}X_2)^{-1}X_2'M_{X_1}X_2(X_2'M_{X_1}X_2)^{-1}X_2'M_{X_1}y}{\sigma^2}\\ &=&\frac{y'M_{X_1}X_2(X_2'M_{X_1}X_2)^{-1}X_2'M_{X_1}y}{\sigma^2}\\ &=&\frac{y'P_{M_{X_1}X_2}y}{\sigma^2}\\ &=:&\frac{y'P_{X_{2\bot1}}y}{\sigma^2}, \end{eqnarray*} where the third equality follows from the Frisch-Waugh-Lovell theorem. Here, $M_A$ and $P_A$ denote the usual residual maker and projection matrices on $A$.

We now give an expression for the Likelihood ratio test statistic under known error variances.

Inserting the restricted and unrestricted estimators, denoted $\widehat{\beta}$ and $\widehat{\beta}_R$, into the sample log-likelihood yields, using \begin{eqnarray*} L(\widehat{\beta})&=&-\frac{n}{2}\log\left(2\pi\sigma^2\right)-\frac{(y-X\widehat{\beta})'(y-X\widehat{\beta})}{2\sigma^2} \end{eqnarray*} and analogously for $L(\widehat{\beta}_R)$, the following expression for the $\mathcal{L}\mathcal{R}$-test statistic: \begin{eqnarray*}\mathcal{L}\mathcal{R}&=&2[L(\widehat{\theta})-L(\widehat{\theta}_R)]\\ &=&\frac{(y-X\widehat{\beta}_R)'(y-X\widehat{\beta}_R)-(y-X\widehat{\beta})'(y-X\widehat{\beta})}{\sigma^2}\\&=&\frac{y'(I-P_{X_1})y-y'(I-P_{X})y}{\sigma^2} \end{eqnarray*}

We now show that $\mathcal{W}$ may also be written in this format, \begin{eqnarray*} \mathcal{W}&=&\frac{y'(I-P_{X_1})y-y'(I-P_{X})y}{\sigma^2} \end{eqnarray*} We first show that $$ P_{X}=P_{X_1}+P_{X_{2\bot1}}, $$ as a partition of $X$, $$X=(X_{A}\vdots X_{B}),$$ in orthogonal matrices $X_{A}$, $X_{B}$ ($X_{A}'X_{B}=0$) satisfies that $$ P_{X}=P_{A}+P_{B} $$ This is so because \begin{eqnarray*} P_{X}&=&\left(\begin{array}{cc} X_{A} & X_{B} \\ \end{array} \right) \left( \begin{array}{cc} X_{A}'X_{A} & X_{A}'X_{B} \\ X_{B}'X_{A} & X_{B}'X_{B} \\ \end{array} \right)^{-1} \left( \begin{array}{c} X_{A}' \\ X_{B}' \\ \end{array} \right)\\ &=&\left(\begin{array}{cc} X_{A} & X_{B} \\ \end{array} \right) \left( \begin{array}{cc} X_{A}'X_{A} & 0 \\ 0 & X_{B}'X_{B} \\ \end{array} \right)^{-1} \left( \begin{array}{c} X_{A}' \\ X_{B}' \\ \end{array} \right)\\ &=&\left(\begin{array}{cc} X_{A} & X_{B} \\ \end{array} \right) \left( \begin{array}{cc} (X_{A}'X_{A})^{-1} & 0 \\ 0 & (X_{B}'X_{B})^{-1} \\ \end{array} \right) \left( \begin{array}{c} X_{A}' \\ X_{B}' \\ \end{array} \right)\\ &=& X_{A}(X_{A}'X_{A})^{-1}X_{A}'+X_{B}(X_{B}'X_{B})^{-1}X_{B}'\\ &=&P_{A}+P_{B} \end{eqnarray*} We can apply this intermediate result to $X_1$ and $X_{2\bot1}$, as $X_{2\bot1}'X_1=0$. Hence, $$P_{X_{2\bot1}}=P_{X}-P_{X_1}.$$ Adding and subtracting $y'Iy$ in the numerator of the Wald statistic completes the proof.

It may be seen from, e.g., this answer that $\mathcal{W}=t^2$, with $t$ the t-ratio for some single coefficient, when we replace $s^2$ by $\sigma^2$, assumed known. Of course, $\mathcal{W}=t^2$ holds more generally when we need to estimate $\sigma^2$. Then, it is however no longer true that $\mathcal{W}=\mathcal{L}\mathcal{R}$.

It may be established that the score test statistic is also numerically equivalent, which is skipped here for brevity.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.