Skip to main content
6 of 6
added 30 characters in body
Zhanxiong
  • 21.2k
  • 2
  • 46
  • 89

The short answer is: you should NOT expect these two tests are strictly equivalent (or mutually derivable).

To corroborate my point above, we need to review some theory. To this end, let's denote the linear model in matrix form by $y = X\beta + \epsilon$, where $y, \epsilon \in \mathbb{R}^{n}, X \in \mathbb{R}^{n \times (p + 1)}$. We are interested in testing the hypothesis:

$$H_0: \beta_1 = \cdots = \beta_p = 0 \text{ v.s. } H_a: \text{ at least one } \beta_i \text{ is non-zero.} \tag{$\dagger$}\label{0}$$

Call the reduced model when $H_0$ is true $M_0$, and the general model $M$. Furthermore, for a given dataset $(y, X)$, denote the residual sum of squares of $M_0$ and $M$ by $RSS(M_0)$ and $RSS(M)$ respectively.

Let's consider the following two cases: $\epsilon \sim N(0, \sigma^2 I_{(n)})$ and $\epsilon$ is arbitrary error.

Case 1: $\epsilon \sim N(0, \sigma^2 I_{(n)})$.

In this case, it is well-known that the likelihood-ratio test statistic for testing $\eqref{0}$ is given by \begin{align*} \Lambda = n\log\left(\frac{RSS(M_0)}{RSS(M)}\right), \tag{1}\label{1} \end{align*} and the F-test statistic for testing $\eqref{0}$ is given by \begin{align*} F = \frac{\frac{RSS(M_0) - RSS(M)}{p - 1}}{\frac{RSS(M)}{n - p - 1}}. \tag{2}\label{2} \end{align*} Under the normality assumption, we know that $F$ in $\eqref{2}$ has an exact $F_{p - 1, n - p - 1}$ distribution (check this answer for the proof). But does $\Lambda$ in $\eqref{1}$ have an exact $\chi_{p - 1}^2$ distribution? NO! It is just asymptotically $\chi_{p - 1}^2$ distributed. This asymptotic result is a consequence of the famous Wilk's theorem. At the end of this answer, I derived the exact distribution of $\Lambda$, from which we can also see the asymptotic distribution of $\Lambda$ is $\chi_{p - 1}^2$.

In conclusion, in Case 1, the p-value of the $F$-test is exact while the $p$-value of the likelihood-ratio test is approximated. Therefore, they are not directly comparable.

Case 2: make no assumption on the distributional form of $\epsilon$.

In this case, although we can still calculate statistics $\eqref{1}$ and $\eqref{2}$, we really cannot assert what their theoretical null distributions are. The reason that $\eqref{2}$ no longer follows $F_{p - 1, n - p - 1}$ can be understood by going through the proof of why $\eqref{2}$ has $F$-distribution under Case 1 (because without the normality assumption, we cannot guarantee the numerator and the denominator have $\chi^2$ distribution, as well as their independence). The reason that $\eqref{1}$ no longer follows $\chi_{p - 1}^2$ asymptotically because with errors that are not homoscedastic Gaussian, $\eqref{1}$ in general is no longer the log-likelihood ratio statistic for comparing $M_0$ and $M$.

The numerical example you presented clearly falls in Case 2. Based on the above analysis, there is no surprise to see these two tests give different $p$-values. A subtler case is Case 1, for which I did the simulation below to illustrate:

set.seed(1)
sigma <- 0.8
n <- 50
epsilon <- rnorm(n, 0, sigma)
x <- runif(n, -1, 1)
y <- 1 + epsilon

withCov <- lm(y ~ x)
withInt <- lm(y ~ 1)

1 - pchisq(2 * (logLik(withCov) - logLik(withInt)), df = 1)
# returns 0.04640223 

summary(withCov)

Call:
lm(formula = y ~ x)

Residuals:
     Min       1Q   Median       3Q      Max 
-1.80682 -0.38006  0.09737  0.51327  1.05222 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  1.08038    0.09134  11.828 7.86e-16 ***
x            0.33067    0.16610   1.991   0.0522 .  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.6459 on 48 degrees of freedom
Multiple R-squared:  0.07627,   Adjusted R-squared:  0.05703 
F-statistic: 3.963 on 1 and 48 DF,  p-value: 0.05221

It can be seen that the two tests even gave contradictory results at the significance level $\alpha = 0.05$. Clearly, the $F$-test did a better job (as expected), and this empirical result supports my theoretical "Case 1" analysis conclusion -- even under the much stronger (and usually unrealistic) normality assumption, the $F$-test and the likelihood-ratio test still do not concur! Therefore, the first sentence of your post "the t-test and F-test we use to get the significance of our regression results are derived from the likelihood ratio test" is a misconception -- although they are numerically related (e.g., if $\Lambda$ in $\eqref{1}$ is large, then $F$ in $\eqref{2}$ is large too because $\eqref{2}$ is a monotonic function of $\eqref{1}$, see $\eqref{3}$ below), it is by no means to say that (the null distribution of) the $F$-test is "derived" from the likelihood-ratio test!


Exact distribution of $\Lambda$ in Case 1.

It follows by $\eqref{1}$ and $\eqref{2}$ that \begin{align*} F = \frac{n - p - 1}{p - 1}(e^{\Lambda/n} - 1), \tag{3}\label{3} \end{align*} whence the Jacobian of transformation $\eqref{3}$ is \begin{align*} J = \frac{n - p - 1}{n(p - 1)}e^{\lambda/n}. \tag{4}\label{4} \end{align*} Since the pdf of a $F_{p - 1, n - p - 1}$ r.v. is \begin{align*} f_F(x) = \frac{1}{B\left(\frac{p - 1}{2}, \frac{n - p - 1}{2}\right)} \left(\frac{p - 1}{n - p - 1}\right)^{\frac{p - 1}{2}}x^{\frac{p - 1}{2} - 1} \left(1 + \frac{p - 1}{n - p - 1}x\right)^{-\frac{n}{2} + 1}, \tag{5}\label{5} \end{align*} it follows by $\eqref{3}, \eqref{4}$ and $\eqref{5}$ that the pdf of $\Lambda$ is given by \begin{align*} f_\Lambda(\lambda) = \frac{1}{B\left(\frac{p - 1}{2}, \frac{n - p - 1}{2}\right)} e^{-\lambda/2} \left(e^{\lambda/n} - 1\right)^{\frac{p - 1}{2} - 1}\frac{1}{n}e^{\frac{2\lambda}{n}}. \tag{6}\label{6} \end{align*} Evidently, $\eqref{6}$ is different from the pdf of a $\chi_{p - 1}^2$ r.v., whose pdf is \begin{align*} g(\lambda) = \frac{1}{2^{\frac{p - 1}{2}}\Gamma\left(\frac{p - 1}{2}\right)}\lambda^{\frac{p - 1}{2} - 1}e^{-\lambda/2}. \tag{7}\label{7} \end{align*} On the other hand, as $n \to \infty$, $f_\Lambda(\lambda)$ does converge to $g(\lambda)$ point-wisely (the verification is not too hard but tedious, where we need to use $e^x \sim 1 + x$ as $x \to 0$ and $\Gamma(z) \sim \sqrt{2\pi}z^{z - 1/2}e^{-z}$ as $z \to \infty$), showing that $\Lambda \to_d \chi_{p - 1}^2$ as $n \to \infty$, as Wilk's theorem asserted.

Zhanxiong
  • 21.2k
  • 2
  • 46
  • 89