Why do we use an F distribution instead of just chi-squared when testing multiple hypotheses from regression?

Question

After estimating a multiple-regression using ordinary least squares (OLS), the classical test for joint significance of multiple variables is to run an F test using restricted and unrestricted regressions:

$$F = \frac{(SSR_r - SSR_{ur})/q}{SSR_{ur}/(n-k-1)} $$

(where here $SSR$ stands for residual sum of squares, $q$ is number of restrictions, and $k$ is the number of slope terms in the unrestricted regression. Wooldridge notation.)

The information from this test really comes from the numerator, which describes how much of the variation our variables of interest are explaining.

This numerator is distributed as chi-sq.

Why not run this test using a chi-sq distribution and ignore the denominator altogether? Is that not valid?

If it would be valid, why do we instead add the denominator to get an F distribution? Where does this fit into the bigger picture?

Answer 1

Hint

This numerator is distributed as chi-sq.

Almost right, but not quite. The precise statement is that the numerator scaled by the dispersion parameter $\sigma^2$ of error $\varepsilon$ is chi-squared distributed (under $H_0$). Can you see the reason why we do need the denominator now?

If still not, I invite you to reconsider a simpler but essentially the same testing problem: testing the location parameter $H_0: \mu = \mu_0$, given a sample $X_1, \ldots, X_n \sim N(\mu, \sigma^2)$, why would not people just use the test statistic $\bar{X} - \mu_0$, but developed the $t$-statistic $T = \frac{\bar{X} - \mu_0}{\frac{\hat{\sigma}}{\sqrt{n}}}$? Compare the numerator and the denominator of the $F$-statistic with that of the $t$-statistic.

Answer 2

Besides Zhanxiong's formal answer (using the notations followed in this post of mine), I would add a brief heuristic as for the issue in hand.

What we are doing is comparing the two models - one the full model and the other being the reduced model in that we hypothesize $\mathbb E[\mathbf Y]\in\mathcal C(\mathbf X_0)\subset \mathcal C(\mathbf X). $ (Think of $\mathcal C(\mathbf X) $ as the two dimensional subspace of a rectangular table surface and $\mathcal C(\mathbf X_0) $ as the one dimensional edge of the table including the origin. By full model, $\mathbb E[\mathbf Y]$ lies somewhere on the surface whereas by the reduced model, it lies on the edge.)

If the reduced model is true, then $\mathbf{MY}=\mathbf{M_0Y}.$ So, we would need to assess how the two differs. A suitable measure could be the squared length of $(\mathbf{M}-\mathbf M_0)\mathbf Y$ divided by the rank of $\mathbf{M}-\mathbf M_0$ to account for the relative sizes of $\mathcal C(\mathbf M) $ and $\mathcal C(\mathbf M_0)$ providing us (which is the numerator of the F statistic) $\mathbf Y^\top(\mathbf M-\mathbf M_0)\mathbf Y/\operatorname{rank}(\mathbf M-\mathbf M_0). $

If it is large, then the reduced model ought not be true. But how large and large by what measure?

For that, notice $\mathbb E[\mathbf Y^\top(\mathbf M-\mathbf M_0)\mathbf Y/\operatorname{rank}(\mathbf M-\mathbf M_0)]=\sigma^2+\boldsymbol\beta^\top\mathbf X^\top(\mathbf{M}-\mathbf M_0)\mathbf X\boldsymbol\beta/\operatorname{rank}(\mathbf M-\mathbf M_0);$ the last term on the right hand side of the equality, called the noncentrality parameter, determines whether the reduced model is true or not in that if it is larger than $\sigma^2, $ then the reduced model is false. (In the table analogy, if $\mathbb E[\mathbf Y]$ happens to lie far from the edge of $\mathcal C(\mathbf X_0), $ it would mean $\bf MY$ would be far from $\mathbf M_0\mathbf Y$, the farness being determined by $\sigma^2,$ prompting us to conclude the reduced model being false.)

Since, $\sigma^2$ is not known to us generally, we estimate it by $\rm MSE$ that is, $\mathbf Y^\top(\mathbf I-\mathbf M) \mathbf Y/\operatorname{rank}(\mathbf I-\mathbf M) $ (the denominator of the F statistic).

It is for the sake of comparison, we have to invoke the denominator. The whole statistic estimates $1+\boldsymbol\beta^\top\mathbf X^\top(\mathbf{M}-\mathbf M_0)\mathbf X\boldsymbol\beta/\operatorname{rank}(\mathbf M-\mathbf M_0)\sigma^2; $ if relative to $\sigma^2$, $\mathbf X\boldsymbol\beta-\mathbf M_0\mathbf X\boldsymbol\beta$ is large, then the model is plausibly not true. In case, however, $\boldsymbol\beta^\top\mathbf X^\top(\mathbf{M}-\mathbf M_0)\mathbf X\boldsymbol\beta/\operatorname{rank}(\mathbf M-\mathbf M_0)\sigma^2$ is small, that is $\mathbf X\boldsymbol\beta-\mathbf M_0\mathbf X\boldsymbol\beta$ is small relative to $\sigma^2, $ the reduced model would be a feasible approximation, even though it is not correct. (In the table analogy, if $\mathbb E[\mathbf Y]=\mathbf X\boldsymbol\beta$ is near but not on the $\mathcal C(\mathbf X_0) $ edge, then it wouldn't be easy to assess the reduced model, but it won't create any problem either.)

---

Reference:

$\rm[I]$ Plane Answers to Complex Questions: Theory of Linear Models, Ronald Christensen, Springer Science$+$Business, $2011,$ sec. $3.2, $ pp. $55-56.$

Answer 3

In addition to the +1 answers: if the errors are indeed normal, then the size of the F-test is exact, where the chi-square test version relying for its asymptotic distribution on the assumption of a consistent estimator of the error variance only has correct size asymptotically (but this for more general error distributions than the normal).

A little Monte Carlo study to illustrate that the F-test has empirical size very close to the nominal 5%, whereas the chi square test is somewhat oversized for smallish $n=20$:

library(lmtest)
n <- 20
reps <- 5000

sloperegs <- 3 # number of slope regressors, q or k-1 (minus the constant) in the above notation
critical.value.F <- qf(p = .95, df1 = sloperegs, df2 = n-sloperegs-1)
critical.value.chisq <- qchisq(p = .95, df = sloperegs) 

# for the null that none of the slope regrssors matter

# betatilde <- rep(.3, sloperegs) # power study
betatilde <- rep(0, sloperegs)  # size study
Fstat <- chisquarestat <- rep(NA,reps)

for (i in 1:reps){
  u <- rnorm(n)
  Xtilde <- matrix(rnorm(n*sloperegs, mean = 1), ncol=sloperegs) # "fixed regressor paradigm"
  y <- Xtilde%*%betatilde + u
  
  reg <- lm(y~Xtilde)
  Fstat[i] <- waldtest(reg, test="F")$F[2]
  chisquarestat[i] <- waldtest(reg, test="Chisq")$Chisq[2] 
}

> mean(Fstat > critical.value.F) # rejection rate F test
[1] 0.0494

> mean(chisquarestat > critical.value.chisq) # rejection rate chi square test
[1] 0.0846