13
$\begingroup$

t-test for testing whether the mean of a normally distributed sample equals a constant is said to be a Wald test, by estimating the standard deviation of the sample mean by the Fisher information of the normal distribution at the sample mean. But the test statistic in the $t$ test has Student's $t$ distribution, while the test statistic in a Wald test asymptotically has a $\chi^{2}$ distribution. I wonder how to explain that?

In one-way ANOVA, the test statistic is defined as the ratio between between-class variance and within-class variance. I was wondering if it is also a Wald test? But the test statistic in one-way ANOVA has a $F$ distribution, and the test statistic in a Wald test asymptotically has a $\chi^{2}$ distribution. I wonder how to explain that?

$\endgroup$

2 Answers 2

21
$\begingroup$

Consider the following setup. We have a $p$-dimensional parameter vector $\theta$ that specifies the model completely and a maximum-likelihood estimator $\hat{\theta}$. The Fisher information in $\theta$ is denoted $I(\theta)$. What is usually referred to as the Wald statistic is

$$(\hat{\theta} - \theta)^T I(\hat{\theta}) (\hat{\theta} - \theta)$$

where $I(\hat{\theta})$ is the Fisher information evaluated in the maximum-likelihood estimator. Under regularity conditions the Wald statistic follows asymptotically a $\chi^2$-distribution with $p$-degrees of freedom when $\theta$ is the true parameter. The Wald statistic can be used to test a simple hypothesis $H_0 : \theta = \theta_0$ on the entire parameter vector.

With $\Sigma(\theta) = I(\theta)^{-1}$ the inverse Fisher information the Wald test statistic of the hypothesis $H_0 : \theta_1 = \theta_{0,1}$ is $$\frac{(\hat{\theta}_1 - \theta_{0,1})^2}{\Sigma(\hat{\theta})_{ii}}.$$ Its asymptotic distribution is a $\chi^2$-distribution with 1 degrees of freedom.

For the normal model where $\theta = (\mu, \sigma^2)$ is the vector of the mean and the variance parameters, the Wald test statistic of testing if $\mu = \mu_0$ is $$\frac{n(\hat{\mu} - \mu_0)^2}{\hat{\sigma}^2}$$ with $n$ the sample size. Here $\hat{\sigma}^2$ is the maximum-likelihood estimator of $\sigma^2$ (where you divide by $n$). The $t$-test statistic is $$\frac{\sqrt{n}(\hat{\mu} - \mu_0)}{s}$$ where $s^2$ is the unbiased estimator of the variance (where you divide by the $n-1$). The Wald test statistic is almost but not exactly equal to the square of the $t$-test statistic, but they are asymptotically equivalent when $n \to \infty$. The squared $t$-test statistic has an exact $F(1, n-1)$-distribution, which converges to the $\chi^2$-distribution with 1 degrees of freedom for $n \to \infty$.

The same story holds regarding the $F$-test in one-way ANOVA.

$\endgroup$
3
  • $\begingroup$ Thanks! I just found that the t test statistic is constructed directly on likelihood ratio test statistic, not on the Wald test statistic. Is one-way ANOVA directly build on likelihood ratio test? $\endgroup$
    – Tim
    Commented Jun 1, 2013 at 1:19
  • 3
    $\begingroup$ @Tim, the $F$-tests used in ANOVA are equivalent to likelihood ratio tests based on the normal error distribution. $\endgroup$
    – NRH
    Commented Jun 1, 2013 at 9:24
  • $\begingroup$ Thanks! Under the normal statistical model, some also says that the distribution of a slight modification of the Wald test statistic has a F distribution under null. Is that true? I post a question here $\endgroup$
    – Tim
    Commented Jun 1, 2013 at 10:45
17
$\begingroup$

@NRH gave a good theoretical answer, here is one that intends to be simpler, more intuitive.

There is the formal Wald test (described in the answer by NRH), but we also refer to tests that look at the difference between an estimated parameter and its hypothesized value relative to the variation estimated at the estimated parameter as a Wald style test. So the t-test as we usually use it is a Wald Style test even if it is slightly different from the exact Wald test (a difference of $n$ vs. $n-1$ inside a square root). We could even design a Wald style test based on an estimated median minus the hypothesized median divided by a function of the IQR, but I don't know what distribution it would follow, it would be better to use a bootstrap, permutation, or simulated distribution for this test rather than depending on chi-square asymptotics. The F-test for ANOVA fits the general pattern as well, the numerator can be thought of as measuring the difference of the means from an overall mean and the denominator is a measure of the variation.

Also note that if you Square a random variable that follows a t distribution then it will follow an F distribution with 1 df for the numerator and the denominator df will be those from the t distribution. Also note that an F distribution with infinite denominator df is a chi-square distribution. So that means that both the t-statistic (squared) and the F statistic are asymptotically chi-squared just like the Wald statistic. We just use the more exact distribution in practice.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.