# How to prove the square of the t-statistic is the F-statistic in a linear regression without Lagrange multipliers?

My question is essentially identical to this In linear regression, how to prove the equivalence of F-test and t-test? - it was (I believe humbly) erroneously marked as a duplicate of this Prove F test is equal to T test squared. These questions are not duplicates, as linear regression is a generalisation of ANOVA and the calculations are quite different, involving more linear algebra.

My question is also identical to ESL Exercise 3.1. In a multivariate linear regression, the square of the t-statistic for $$H_0: \beta_j=0$$ is $$\frac{\hat{\beta_j}^2}{\hat{\sigma}^2 (X^T X)^{-1}_{jj}}$$

This should coincide with the F-statistic comparing the full model with the model that leaves out the variable $$X_j$$. This quickly reduces to proving

$$RSS_0 - RSS = \frac{\hat{\beta_j}^2}{(X^T X)^{-1}_{jj}}$$

I have seen this problem solved with Lagrange multipliers (eg here https://yuhangzhou88.github.io/ESL_Solution/ESL-Solution/3-Linear-Methods-for-Regression/ex3-01/), but it should be able to be solved purely with linear algebra.

For example, $$RSS= y^T(I-H)y$$ where $$H=X(X^TX)^{-1}X^T$$ is the hat matrix and the projection onto the column space of $$X$$. Similarly, $$RSS_0 = y^T(I-H_0)y$$. On the other hand, $$\hat{\beta}=(X^TX)^{-1}X^Ty$$ so $$\hat{\beta}_j = Row_j((X^TX)^{-1}) X^Ty$$. So

$$\hat{\beta}_j^2 = y^T X Row_j((X^TX)^{-1})^T Row_j((X^TX)^{-1}) X^T y$$

Thus we can remove $$y$$ entirely and just need to prove the following

$$X Row_j((X^TX)^{-1})^T Row_j((X^TX)^{-1}) X^T = (X^T X)^{-1}_{jj} (H - H_0).$$

That is, the desired equality is now something entirely expressible in terms of $$X$$, but I find it quite intractable. It reminds me of Schur complement/inversion, which I have never really understood, so if you use that, an explanation of that would be appreciated too.

Edit: I have found quite a nice proof of this ultimately by careful use of orthogonal decompositions, but I'm still wondering if this has something to do with Schur inversion.

• I have removed the additional query on Cook's D, which would warrant ab additional question, if OP wants. Feb 10 at 4:35

More appealing (than Lagrangian multipliers), intuitive from a geometrical point of view is to resort to perpendicular projection operators.

It is apt to work in a more general setting: let us consider for the linear model $$\mathbf Y = \mathbf X\boldsymbol\beta +\boldsymbol{\varepsilon},$$ the hypothesis $$\mathrm H_0: \mathbf \Lambda^\top\boldsymbol{\beta}= \mathbf 0.\tag 1\label 1$$

By $$\eqref 1,~ \boldsymbol\beta \in \mathcal C(\mathbf \Lambda)^{\perp}=:\mathcal C(\mathbf U)\implies \boldsymbol\beta = \mathbf U\boldsymbol\gamma.$$ Using this, we can get the reduced model as $$\mathbf Y = \mathbf X_0\boldsymbol\gamma +\boldsymbol{\varepsilon}, ~~\mathbf X_0:= \mathbf {XU}.$$

We know the numerator of the F-statistic is expressed as $$\mathbf Y^\top(\mathbf M-\mathbf M_0)\mathbf Y/\operatorname{rank}(\mathbf M-\mathbf M_0),\tag 2\label 2$$ where $$\mathbf M,~ \mathbf M_0$$ are respectively the perpendicular projection operators onto $$\mathcal C(\mathbf X)$$ and $$\mathcal C(\mathbf X_0) .$$ It is a routine calculation to check $$\mathcal C(\mathbf M-\mathbf M_0)= \mathcal C(\mathbf X_0)^{\perp}_{\mathcal C(\mathbf X)}.$$

Assume the linear constraint in $$\eqref 1$$ is estimable (If not, we would consider the estimable part of $$\eqref 1: \mathbf \Lambda_0^\top\boldsymbol{\beta}= \mathbf 0,$$ where $$\mathcal C(\boldsymbol\Lambda_0):= \mathcal C(\boldsymbol\Lambda)\cap \mathcal C\left(\mathbf X^\top\right);$$ the rationale behind it being that $$\mathcal C(\boldsymbol\Lambda_0)\subset \mathcal C(\boldsymbol\Lambda)\implies \mathbf \Lambda_0^\top\boldsymbol{\beta}= \mathbf 0$$ and $$\mathcal C(\boldsymbol\Lambda_0)\subset\mathcal C\left(\mathbf X^\top\right)$$ means $$\mathbf \Lambda_0^\top\boldsymbol{\beta}$$ is estimable; both constraints induce the same reduced model). Then $$\mathbf \Lambda = \mathbf X^\top\mathbf P,$$ for some $$\mathbf P.$$

The goal is to re-write $$\eqref 2$$ in terms of $$\mathbf \Lambda$$ and $$\hat{\boldsymbol\beta}.$$ For that, the approach would be to express $$\mathbf M-\mathbf M_0$$ in terms of $$\mathbf \Lambda.$$

Result $$1.$$ $$\mathcal C(\mathbf M-\mathbf M_0) =\mathcal C(\mathbf{MP} ).$$

From above $$\mathcal C(\mathbf M-\mathbf M_0)= \mathcal C(\mathbf {XU})^{\perp}_{\mathcal C(\mathbf X)}.$$ If $$\mathbf x\in \mathcal C(\mathbf {XU})^{\perp}_{\mathcal C(\mathbf X)},$$ then $$\mathbf X^\top\mathbf x\perp \mathcal C(\mathbf U)\implies \mathbf X^\top\mathbf x\in \mathcal C(\mathbf \Lambda).$$ Now $$\mathbf x = \mathbf{ Mx} = \mathbf X\left(\mathbf X^\top \mathbf X\right)^{-}\mathbf X^\top\mathbf x\in \mathcal C(\mathbf{MP} ).$$ Conversely, $$\mathbf x\in \mathcal C(\mathbf{MP})\implies \mathbf x= \mathbf{MPb},$$ and $$\mathbf x^\top\mathbf {XU} = \mathbf b^\top\underbrace{\mathbf P^\top\underbrace{\mathbf M\mathbf X}_{\mathbf{MX}= \mathbf X}}_{\mathbf P^\top\mathbf X = \mathbf \Lambda^\top }\mathbf U = 0,$$ as $$\mathcal C(\mathbf U)\equiv \mathcal C(\mathbf \Lambda)^{\perp}.$$

$$\blacksquare$$

Result $$2.$$ $$\operatorname{rank}(\mathbf \Lambda) =\operatorname{rank}\left(\mathbf M_{\mathbf{ MP}}\right).$$

It suffices to check $$\mathcal C(\mathbf X^\top\mathbf P) = \mathcal C(\mathbf P^\top\mathbf M)$$ as this would imply $$\operatorname{rank}(\mathbf X^\top\mathbf P) = \operatorname{rank}\left(\left(\mathbf P^\top\mathbf M\right)^\top\right)= \operatorname{rank}\left(\mathbf{ MP}\right)=\operatorname{rank}\left(\mathbf M_{\mathbf{ MP}}\right) .$$ Now, $$\mathbf X^\top\mathbf P\mathbf z = 0\iff \mathbf P\mathbf z\perp \mathcal C(\mathbf X) = \mathcal C(\mathbf M).$$ This means $$\mathcal C(\mathbf X^\top\mathbf P)^\perp = \mathcal C(\mathbf P^\top\mathbf M)^\perp.$$

$$\blacksquare$$

Using these two results, we can re-write $$\eqref 2$$ as

\begin{align}\frac{\mathbf Y^\top(\mathbf M-\mathbf M_0)\mathbf Y}{\operatorname{rank}(\mathbf M-\mathbf M_0)} &= \frac{\mathbf Y^\top\mathbf M_\mathbf{MP}\mathbf Y}{\operatorname{rank}(\mathbf \Lambda)}\\&=\frac{\mathbf Y^\top\mathbf{MP}\left(\mathbf P^\top\mathbf{MP}\right)^{-}\mathbf P^\top\mathbf{M Y}}{\operatorname{rank}(\mathbf \Lambda)} \\&=\frac{\hat{\boldsymbol \beta}^\top\mathbf{X}^\top\mathbf {P}\left(\mathbf P^\top\mathbf X\left(\mathbf X^\top\mathbf X\right)^{-}\mathbf X^\top\mathbf{P}\right)^{-}\mathbf P^\top\mathbf{X}\hat{\boldsymbol\beta}}{\operatorname{rank}(\mathbf \Lambda)}\\&=\frac{\hat{\boldsymbol \beta}^\top\mathbf{\Lambda}\left(\mathbf \Lambda^\top\left(\mathbf X^\top\mathbf X\right)^{-}\mathbf \Lambda\right)^{-}\mathbf{ \Lambda}^\top\hat{\boldsymbol\beta}}{\operatorname{rank}(\mathbf \Lambda)}\tag 3,\label 3\end{align}

which is the desired form.

If $$\eqref 1$$ is of single degree of freedom hypothesis, say, $$\mathrm H_0: \boldsymbol \lambda^\top\boldsymbol{\beta}= 0,$$ then $$\eqref 3$$ becomes $$\frac{\left(\boldsymbol\lambda^\top\hat{\boldsymbol\beta}\right)^2}{\left(\boldsymbol\lambda^\top\left(\mathbf X^\top\mathbf X\right)^{-}\boldsymbol\lambda\right)}.$$

$$\rm[II]$$ also has somewhat similar treatment using perpendicular projection operators specifically for multiple regression problems.

## References:

$$\rm[I]$$ Plane Answers to Complex Questions: Theory of Linear Models, Ronald Christensen, Springer Science$$+$$Business, $$2011,$$ sec. $$3.2-3.3.$$

$$\rm[II]$$ Linear Regression Analysis, George A. F. Seber, Alan J. Lee, John Wiley & Sons, $$2003,$$ sec. $$3.8.2, ~4.3.2.$$

• I really appreciate the explicitness of the mathematics in this answer (+1). Feb 10 at 5:01
• Thanks, this looks similar in scope to the orthogonal projections proof that I found, but certainly in a bit of a different direction. I'm still curious if there is a relation with Schur's inversion. Feb 18 at 5:40
• By Schur's inversion, are you talking about the Schur formulas for calculating inverse of partitioned matrices? AFAIR, it is not directly needed in the derivation of the result. Feb 19 at 2:25