MLE-based hypothesis tests

Question

I recently encountered the three MLE-based tests Wald test, Likelihood ratio test and Lagrange Multiplier test. Although it seemed at first like the usual hypothesis testing I already know from statistics, I got in some trouble with regard to the actual application. In an attempt to fully understand all three I got my hands on some practice problems. Since all tests are built on MLE, asymptotically equivalent and the points I don't get are similar for all three, I will just ask my question to all jointly here. Hence the post will be longer, apologies. If I should pose them separately, I will, of course, do so on request.

First, the Wald test. Take for example 100 realizations of a normally distributed random variable with mean $\mu = 0.36$ and $\sigma^2 = 4$. Define $c_1(\mu) = \mu$. How would you conduct a Wald test for $H_0: c(\mu) = c_1(\mu) - 0.8 = 0$ at the 5% significance level practically? Also, what I don't understand is why one usually defines another function $c_1(\mu)$? What I understood here, is that one tests the restriction $$c(\mu) = 0$$ and the closer the value of the test statistic as well as the value of $c(\mu)$ is to zero the more valid the restriction. Since it is normally distributed one can set $\mu$ as the MLE estimate for the mean. But how does it go from here exactly? I calculate $$ c(0.36) = 0.36 - 0.8 = -0.44$$

and, according to my materials,

$$W = [-0.44](Var(-0.44))^{-1}(-0.44)$$

But what is $Var(-0.44)$? And what is the underlying distribution from which I get the p-value?

Second, the likelihood ratio test. Take again a sample of 100 observations. But this time from the poisson distribution. The sample mean here is $\mu = 1.7$. Therefore, the MLE estimate of $\lambda$ is also $\hat{\lambda} = 1.7$. This time consider $$c(\lambda) = \lambda^2 - 3\lambda + 2$$ How to test for $c(\lambda) = 0$ at the 5% level? Here, I understand even less how to get along with so little information since I thought that one would need to evaluate the log-likelihood function and then decide based on the difference between the log-likelihood value of the restricted and unrestricted MLE estimate? And again, which distribution does the statistic (and thus the p-value) follow?

Finally, the Lagrange Multiplier test. I thought here as well that I would need the log-likelihood function since I have to insert the restricted estimate in its derivative, don't I? Take the same distribution as before but with the function $$c(\lambda) = \frac{1}{\lambda^2} - 0.1$$ What is the restricted MLE estimate that I insert in the log-likelihood derivative? Is it $\frac{1}{1.7^2} - 0.1$? How do I go about it without having the actual sample and the log-likelihood function at my disposal?

Answer 1

Quite often when people start out working through this stuff they have difficulties because they don't keep clear track of which symbols refer to parameters and which symbols refer to parameter estimates (which are functions of the data). I think that's what happened to you!

I'll address the three tests in turn.

The Wald test is performed by calculating the squared difference between the MLE and the hypothesized value of the parameter, divided by the inverse of the expected Fisher information evaluated at the MLE (basically, an estimate of the variance of the MLE). This can be written for a generic parameter $\theta$ as:

$W = (\hat{\theta}-\theta)^2 / I^{-1}(\hat{\theta})$

The Fisher information is defined as:

$I(\theta) = \int (d \log f(x;\theta)/d\theta)^2 f(x;\theta)dx$

or, when the second derivative exists and certain regularity conditions are met:

$I(\theta) = \int (d^2 \log f(x;\theta)/d\theta^2) f(x;\theta)dx$

Often finding the second derivative of the log likelihood function and substituting in the expected values of the various functions of $x$ that appear is straightforward, for example, in the case of the Poisson distribution the only function of $x$ that appears is $\sum_{i=1}^n x_i$, which is easily seen to have expectation $n\lambda$.

As, under the null hypothesis, the MLE is asymptotically distributed Normal with mean = whatever the hypothesized value is, the asymptotic distribution of the test statistic is $\chi^2$ (issues of regularity aside.)

In your example, I will assume you were a little loose with your description, and that your sample mean $\hat{\mu} = 0.36$, your sample standard deviation $\hat{\sigma} = 4$, and your sample size $n=100$. For the Normal distribution, the inverse of the expected Fisher information evaluated at the MLE is just $\hat{\sigma}^2/n$. Your test of the hypothesis $\mu = 0.8$ would result in the Wald statistic $W = (0.36-0.8)^2 / (4^2/100) = 1.21$. We would compare this to the critical values of a $\chi^2(1)$ distribution, and observe that we cannot reject the null hypothesis at any meaningful level of confidence.

The LR test is computed by calculating the test statistic:

$\Lambda = -2 (l(\theta) - l(\hat{\theta}))$

where $l$ is the log likelihood function. Note that you evaluate $l$ twice, once at the MLE and once at the hypothesized value of the parameter, but that you don't have to do this for the Wald test. Asymptotically, if the null hypothesis is true, this test statistic also has a $\chi^2$ distribution, with the number of degrees of freedom equal to the number of parameters with specified values under the null hypothesis.

Under your null hypothesis, $\lambda = 2$ (solve the quadratic equation and note that $\lambda$ must be positive.) We observe that the log of the likelihood function, up to a constant which will cancel out when calculating the test statistic, is $l = -n\lambda + n\bar{x}\ln \lambda$. Substituting $2$ and $1.7$ in as is appropriate for the hypothesized value of $\lambda$ and the MLE of $\lambda$ respectively, and also using the sample mean $\bar{x} = 1.7$, we calculate:

$l(2) = -100*2 + 100*1.7*\log(2) = -82.16$ $l(1.7) = -100*1.7 + 100*1.7*\log(1.7) = -79.79$ $\Lambda = -2*(-82.16-(-79.79)) = 4.76$

Comparing to the $\chi^2(1)$ distribution, we see that we could reject the null hypothesis at the 95% level of confidence but not the 99% level of confidence.

The Lagrange Multiplier test, also known as the score test, is a lot like the Wald test, in that you don't evaluate the likelihood twice and you use the Fisher information matrix for calculation of a variance. You evaluate the square of the derivative of the log likelihood at the hypothesized value and divide by the Fisher information, again evaluated at the null hypothesis:

$LM = \frac{(d l(\theta)/d \theta)^2}{I(\theta)}$

Note that in the Wald test the Fisher information is evaluated at the MLE, not the null hypothesis, and that we divide by its inverse. Again, asymptotically, if the null hypothesis is true, this test statistic has a $\chi^2$ distribution with degrees of freedom equal to the number of parameters being tested. In many cases the derivative of the log likelihood will be proportional to the difference between the MLE and the hypothesized parameter value, making the test strongly resemble the Wald test.

You cannot do this calculation without the sample, or sufficient statistics thereof, and the log likelihood function, since the likelihood function needs data to be able to calculate a value! This is how the data gets into the test - through the likelihood function, rather than through an explicit parameter estimate appearing in the formulation of the test itself.

In your case, working through the algebra of $c(\lambda)$ results in a null hypothesis of $\lambda = \sqrt{10}$. I'll toss this out and use the same null hypothesis as with the LR test, as that will be a little more educational.

Let's do a little algebra. We'll start with the derivative of the log likelihood $l = -n\lambda + \sum{x_i}\ln \lambda$:

$\text{d}l/\text{d}\lambda = -n + \sum x_i/\lambda = -n + n\hat{\lambda}/\lambda = n(\hat{\lambda}-\lambda)/\lambda$

Squaring this and dividing by $n/\lambda$ (the Fisher information) gives:

$LM = (\hat{\lambda}-\lambda)^2 / (\lambda/n)$.

If we were to construct a Wald test for this problem, the numerator would be the squared difference between the MLE and the hypothesized value of the parameter $(\hat{\lambda}-\lambda)^2$, and the Fisher information would be evaluated at the MLE $\hat{\lambda}$. Consequently, the test statistic would be:

$W = (\hat{\lambda}-\lambda)^2 / (\hat{\lambda}/n)$.

Substituting in the various values in your example gives the Lagrange Multiplier and Wald test statistics $LM = 4.5$, $W = 5.29$. In both of these cases, we would be able to reject the null hypothesis of $\lambda = 2$ at the 95% but not the 99% level of confidence, as with the Likelihood Ratio test.

Of course, we could have substituted values directly into the initial expression for $\text{d} l/\text{d}\lambda$ and we would have gotten the same answer, but working through the algebra does allow us to observe a not uncommon relationship between the LM and Wald tests, namely, that the numerator is often the same with the denominator only differing in that the LM test uses the null hypothesis parameters and the Wald test uses the MLEs of the parameters.

For these examples, the calculations are pretty simple, because the distributions are simple and we are assuming the observations are independent and identically distributed. If our observations are not independent, typically the calculations become more complex, often far more so, and the number of parameters that need to be estimated can increase substantially too. This also changes a lot of the calculations to matrix calculations instead of scalar calculations, e.g., the Fisher information becomes a matrix instead of a scalar, and expressions of the form $\hat{\lambda} - \lambda$ become vectors rather than scalars, resulting in, for example, $W = (\hat{\theta}-\theta)^TI(\theta)(\hat{\theta}-\theta)$. The expressions also increase in complexity if we consider tests on functions of the parameters rather than directly on the parameter values themselves.