0
$\begingroup$

My question is essentially identical to this In linear regression, how to prove the equivalence of F-test and t-test? - it was (I believe humbly) erroneously marked as a duplicate of this Prove F test is equal to T test squared. These questions are not duplicates, as linear regression is a generalisation of ANOVA and the calculations are quite different, involving more linear algebra.

My question is also identical to ESL Exercise 3.1. In a multivariate linear regression, the square of the t-statistic for $H_0: \beta_j=0$ is $$\frac{\hat{\beta_j}^2}{\hat{\sigma}^2 (X^T X)^{-1}_{jj}}$$

This should coincide with the F-statistic comparing the full model with the model that leaves out the variable $X_j$. This quickly reduces to proving

$$RSS_0 - RSS = \frac{\hat{\beta_j}^2}{(X^T X)^{-1}_{jj}}$$

I have seen this problem solved with Lagrange multipliers (eg here https://yuhangzhou88.github.io/ESL_Solution/ESL-Solution/3-Linear-Methods-for-Regression/ex3-01/), but it should be able to be solved purely with linear algebra.

For example, $RSS= y^T(I-H)y$ where $H=X(X^TX)^{-1}X^T$ is the hat matrix and the projection onto the column space of $X$. Similarly, $RSS_0 = y^T(I-H_0)y$. On the other hand, $\hat{\beta}=(X^TX)^{-1}X^Ty$ so $\hat{\beta}_j = Row_j((X^TX)^{-1}) X^Ty$. So

$$ \hat{\beta}_j^2 = y^T X Row_j((X^TX)^{-1})^T Row_j((X^TX)^{-1}) X^T y $$

Thus we can remove $y$ entirely and just need to prove the following

$$X Row_j((X^TX)^{-1})^T Row_j((X^TX)^{-1}) X^T = (X^T X)^{-1}_{jj} (H - H_0). $$

That is, the desired equality is now something entirely expressible in terms of $X$, but I find it quite intractable. It reminds me of Schur complement/inversion, which I have never really understood, so if you use that, an explanation of that would be appreciated too.

Edit: I have found quite a nice proof of this ultimately by careful use of orthogonal decompositions, but I'm still wondering if this has something to do with Schur inversion.

$\endgroup$
1
  • $\begingroup$ I have removed the additional query on Cook's D, which would warrant ab additional question, if OP wants. $\endgroup$ Feb 10 at 4:35

1 Answer 1

2
$\begingroup$

More appealing (than Lagrangian multipliers), intuitive from a geometrical point of view is to resort to perpendicular projection operators.

It is apt to work in a more general setting: let us consider for the linear model $$\mathbf Y = \mathbf X\boldsymbol\beta +\boldsymbol{\varepsilon},$$ the hypothesis $$\mathrm H_0: \mathbf \Lambda^\top\boldsymbol{\beta}= \mathbf 0.\tag 1\label 1$$

By $\eqref 1,~ \boldsymbol\beta \in \mathcal C(\mathbf \Lambda)^{\perp}=:\mathcal C(\mathbf U)\implies \boldsymbol\beta = \mathbf U\boldsymbol\gamma.$ Using this, we can get the reduced model as $\mathbf Y = \mathbf X_0\boldsymbol\gamma +\boldsymbol{\varepsilon}, ~~\mathbf X_0:= \mathbf {XU}.$

We know the numerator of the F-statistic is expressed as $$\mathbf Y^\top(\mathbf M-\mathbf M_0)\mathbf Y/\operatorname{rank}(\mathbf M-\mathbf M_0),\tag 2\label 2$$ where $\mathbf M,~ \mathbf M_0$ are respectively the perpendicular projection operators onto $\mathcal C(\mathbf X) $ and $\mathcal C(\mathbf X_0) .$ It is a routine calculation to check $\mathcal C(\mathbf M-\mathbf M_0)= \mathcal C(\mathbf X_0)^{\perp}_{\mathcal C(\mathbf X)}.$

Assume the linear constraint in $\eqref 1$ is estimable (If not, we would consider the estimable part of $\eqref 1: \mathbf \Lambda_0^\top\boldsymbol{\beta}= \mathbf 0,$ where $\mathcal C(\boldsymbol\Lambda_0):= \mathcal C(\boldsymbol\Lambda)\cap \mathcal C\left(\mathbf X^\top\right);$ the rationale behind it being that $\mathcal C(\boldsymbol\Lambda_0)\subset \mathcal C(\boldsymbol\Lambda)\implies \mathbf \Lambda_0^\top\boldsymbol{\beta}= \mathbf 0$ and $\mathcal C(\boldsymbol\Lambda_0)\subset\mathcal C\left(\mathbf X^\top\right)$ means $\mathbf \Lambda_0^\top\boldsymbol{\beta}$ is estimable; both constraints induce the same reduced model). Then $\mathbf \Lambda = \mathbf X^\top\mathbf P,$ for some $\mathbf P.$

The goal is to re-write $\eqref 2$ in terms of $\mathbf \Lambda$ and $\hat{\boldsymbol\beta}.$ For that, the approach would be to express $\mathbf M-\mathbf M_0$ in terms of $\mathbf \Lambda.$

Result $1.$ $$\mathcal C(\mathbf M-\mathbf M_0) =\mathcal C(\mathbf{MP} ).$$

From above $\mathcal C(\mathbf M-\mathbf M_0)= \mathcal C(\mathbf {XU})^{\perp}_{\mathcal C(\mathbf X)}.$ If $\mathbf x\in \mathcal C(\mathbf {XU})^{\perp}_{\mathcal C(\mathbf X)},$ then $\mathbf X^\top\mathbf x\perp \mathcal C(\mathbf U)\implies \mathbf X^\top\mathbf x\in \mathcal C(\mathbf \Lambda).$ Now $$\mathbf x = \mathbf{ Mx} = \mathbf X\left(\mathbf X^\top \mathbf X\right)^{-}\mathbf X^\top\mathbf x\in \mathcal C(\mathbf{MP} ).$$ Conversely, $\mathbf x\in \mathcal C(\mathbf{MP})\implies \mathbf x= \mathbf{MPb},$ and $$\mathbf x^\top\mathbf {XU} = \mathbf b^\top\underbrace{\mathbf P^\top\underbrace{\mathbf M\mathbf X}_{\mathbf{MX}= \mathbf X}}_{\mathbf P^\top\mathbf X = \mathbf \Lambda^\top }\mathbf U = 0, $$ as $\mathcal C(\mathbf U)\equiv \mathcal C(\mathbf \Lambda)^{\perp}.$

$\blacksquare$

Result $2.$ $$\operatorname{rank}(\mathbf \Lambda) =\operatorname{rank}\left(\mathbf M_{\mathbf{ MP}}\right). $$

It suffices to check $\mathcal C(\mathbf X^\top\mathbf P) = \mathcal C(\mathbf P^\top\mathbf M)$ as this would imply $\operatorname{rank}(\mathbf X^\top\mathbf P) = \operatorname{rank}\left(\left(\mathbf P^\top\mathbf M\right)^\top\right)= \operatorname{rank}\left(\mathbf{ MP}\right)=\operatorname{rank}\left(\mathbf M_{\mathbf{ MP}}\right) . $ Now, $\mathbf X^\top\mathbf P\mathbf z = 0\iff \mathbf P\mathbf z\perp \mathcal C(\mathbf X) = \mathcal C(\mathbf M).$ This means $\mathcal C(\mathbf X^\top\mathbf P)^\perp = \mathcal C(\mathbf P^\top\mathbf M)^\perp.$

$\blacksquare$

Using these two results, we can re-write $\eqref 2$ as

$$\begin{align}\frac{\mathbf Y^\top(\mathbf M-\mathbf M_0)\mathbf Y}{\operatorname{rank}(\mathbf M-\mathbf M_0)} &= \frac{\mathbf Y^\top\mathbf M_\mathbf{MP}\mathbf Y}{\operatorname{rank}(\mathbf \Lambda)}\\&=\frac{\mathbf Y^\top\mathbf{MP}\left(\mathbf P^\top\mathbf{MP}\right)^{-}\mathbf P^\top\mathbf{M Y}}{\operatorname{rank}(\mathbf \Lambda)} \\&=\frac{\hat{\boldsymbol \beta}^\top\mathbf{X}^\top\mathbf {P}\left(\mathbf P^\top\mathbf X\left(\mathbf X^\top\mathbf X\right)^{-}\mathbf X^\top\mathbf{P}\right)^{-}\mathbf P^\top\mathbf{X}\hat{\boldsymbol\beta}}{\operatorname{rank}(\mathbf \Lambda)}\\&=\frac{\hat{\boldsymbol \beta}^\top\mathbf{\Lambda}\left(\mathbf \Lambda^\top\left(\mathbf X^\top\mathbf X\right)^{-}\mathbf \Lambda\right)^{-}\mathbf{ \Lambda}^\top\hat{\boldsymbol\beta}}{\operatorname{rank}(\mathbf \Lambda)}\tag 3,\label 3\end{align}$$

which is the desired form.

If $\eqref 1$ is of single degree of freedom hypothesis, say, $\mathrm H_0: \boldsymbol \lambda^\top\boldsymbol{\beta}= 0,$ then $\eqref 3$ becomes $\frac{\left(\boldsymbol\lambda^\top\hat{\boldsymbol\beta}\right)^2}{\left(\boldsymbol\lambda^\top\left(\mathbf X^\top\mathbf X\right)^{-}\boldsymbol\lambda\right)}.$

$\rm[II]$ also has somewhat similar treatment using perpendicular projection operators specifically for multiple regression problems.


References:

$\rm[I]$ Plane Answers to Complex Questions: Theory of Linear Models, Ronald Christensen, Springer Science$+$Business, $2011,$ sec. $3.2-3.3.$

$\rm[II]$ Linear Regression Analysis, George A. F. Seber, Alan J. Lee, John Wiley & Sons, $2003,$ sec. $3.8.2, ~4.3.2.$

$\endgroup$
3
  • 1
    $\begingroup$ I really appreciate the explicitness of the mathematics in this answer (+1). $\endgroup$
    – Galen
    Feb 10 at 5:01
  • $\begingroup$ Thanks, this looks similar in scope to the orthogonal projections proof that I found, but certainly in a bit of a different direction. I'm still curious if there is a relation with Schur's inversion. $\endgroup$ Feb 18 at 5:40
  • $\begingroup$ By Schur's inversion, are you talking about the Schur formulas for calculating inverse of partitioned matrices? AFAIR, it is not directly needed in the derivation of the result. $\endgroup$ Feb 19 at 2:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.