1
$\begingroup$

I recently encountered the three MLE-based tests Wald test, Likelihood ratio test and Lagrange Multiplier test. Although it seemed at first like the usual hypothesis testing I already know from statistics, I got in some trouble with regard to the actual application. In an attempt to fully understand all three I got my hands on some practice problems. Since all tests are built on MLE, asymptotically equivalent and the points I don't get are similar for all three, I will just ask my question to all jointly here. Hence the post will be longer, apologies. If I should pose them separately, I will, of course, do so on request.

First, the Wald test. Take for example 100 realizations of a normally distributed random variable with mean $\mu = 0.36$ and $\sigma^2 = 4$. Define $c_1(\mu) = \mu$. How would you conduct a Wald test for $H_0: c(\mu) = c_1(\mu) - 0.8 = 0$ at the 5% significance level practically? Also, what I don't understand is why one usually defines another function $c_1(\mu)$? What I understood here, is that one tests the restriction $$c(\mu) = 0$$ and the closer the value of the test statistic as well as the value of $c(\mu)$ is to zero the more valid the restriction. Since it is normally distributed one can set $\mu$ as the MLE estimate for the mean. But how does it go from here exactly? I calculate $$ c(0.36) = 0.36 - 0.8 = -0.44$$

and, according to my materials,

$$W = [-0.44](Var(-0.44))^{-1}(-0.44)$$

But what is $Var(-0.44)$? And what is the underlying distribution from which I get the p-value?

Second, the likelihood ratio test. Take again a sample of 100 observations. But this time from the poisson distribution. The sample mean here is $\mu = 1.7$. Therefore, the MLE estimate of $\lambda$ is also $\hat{\lambda} = 1.7$. This time consider $$c(\lambda) = \lambda^2 - 3\lambda + 2$$ How to test for $c(\lambda) = 0$ at the 5% level? Here, I understand even less how to get along with so little information since I thought that one would need to evaluate the log-likelihood function and then decide based on the difference between the log-likelihood value of the restricted and unrestricted MLE estimate? And again, which distribution does the statistic (and thus the p-value) follow?

Finally, the Lagrange Multiplier test. I thought here as well that I would need the log-likelihood function since I have to insert the restricted estimate in its derivative, don't I? Take the same distribution as before but with the function $$c(\lambda) = \frac{1}{\lambda^2} - 0.1$$ What is the restricted MLE estimate that I insert in the log-likelihood derivative? Is it $\frac{1}{1.7^2} - 0.1$? How do I go about it without having the actual sample and the log-likelihood function at my disposal?

$\endgroup$
2
+50
$\begingroup$

Quite often when people start out working through this stuff they have difficulties because they don't keep clear track of which symbols refer to parameters and which symbols refer to parameter estimates (which are functions of the data). I think that's what happened to you!

I'll address the three tests in turn.

The Wald test is performed by calculating the squared difference between the MLE and the hypothesized value of the parameter, divided by the inverse of the expected Fisher information evaluated at the MLE (basically, an estimate of the variance of the MLE). This can be written for a generic parameter $\theta$ as:

$W = (\hat{\theta}-\theta)^2 / I^{-1}(\hat{\theta})$

The Fisher information is defined as:

$I(\theta) = \int (d \log f(x;\theta)/d\theta)^2 f(x;\theta)dx$

or, when the second derivative exists and certain regularity conditions are met:

$I(\theta) = \int (d^2 \log f(x;\theta)/d\theta^2) f(x;\theta)dx$

Often finding the second derivative of the log likelihood function and substituting in the expected values of the various functions of $x$ that appear is straightforward, for example, in the case of the Poisson distribution the only function of $x$ that appears is $\sum_{i=1}^n x_i$, which is easily seen to have expectation $n\lambda$.

As, under the null hypothesis, the MLE is asymptotically distributed Normal with mean = whatever the hypothesized value is, the asymptotic distribution of the test statistic is $\chi^2$ (issues of regularity aside.)

In your example, I will assume you were a little loose with your description, and that your sample mean $\hat{\mu} = 0.36$, your sample standard deviation $\hat{\sigma} = 4$, and your sample size $n=100$. For the Normal distribution, the inverse of the expected Fisher information evaluated at the MLE is just $\hat{\sigma}^2/n$. Your test of the hypothesis $\mu = 0.8$ would result in the Wald statistic $W = (0.36-0.8)^2 / (4^2/100) = 1.21$. We would compare this to the critical values of a $\chi^2(1)$ distribution, and observe that we cannot reject the null hypothesis at any meaningful level of confidence.

The LR test is computed by calculating the test statistic:

$\Lambda = -2 (l(\theta) - l(\hat{\theta}))$

where $l$ is the log likelihood function. Note that you evaluate $l$ twice, once at the MLE and once at the hypothesized value of the parameter, but that you don't have to do this for the Wald test. Asymptotically, if the null hypothesis is true, this test statistic also has a $\chi^2$ distribution, with the number of degrees of freedom equal to the number of parameters with specified values under the null hypothesis.

Under your null hypothesis, $\lambda = 2$ (solve the quadratic equation and note that $\lambda$ must be positive.) We observe that the log of the likelihood function, up to a constant which will cancel out when calculating the test statistic, is $l = -n\lambda + n\bar{x}\ln \lambda$. Substituting $2$ and $1.7$ in as is appropriate for the hypothesized value of $\lambda$ and the MLE of $\lambda$ respectively, and also using the sample mean $\bar{x} = 1.7$, we calculate:

$l(2) = -100*2 + 100*1.7*\log(2) = -82.16$ $l(1.7) = -100*1.7 + 100*1.7*\log(1.7) = -79.79$ $\Lambda = -2*(-82.16-(-79.79)) = 4.76$

Comparing to the $\chi^2(1)$ distribution, we see that we could reject the null hypothesis at the 95% level of confidence but not the 99% level of confidence.

The Lagrange Multiplier test, also known as the score test, is a lot like the Wald test, in that you don't evaluate the likelihood twice and you use the Fisher information matrix for calculation of a variance. You evaluate the square of the derivative of the log likelihood at the hypothesized value and divide by the Fisher information, again evaluated at the null hypothesis:

$LM = \frac{(d l(\theta)/d \theta)^2}{I(\theta)}$

Note that in the Wald test the Fisher information is evaluated at the MLE, not the null hypothesis, and that we divide by its inverse. Again, asymptotically, if the null hypothesis is true, this test statistic has a $\chi^2$ distribution with degrees of freedom equal to the number of parameters being tested. In many cases the derivative of the log likelihood will be proportional to the difference between the MLE and the hypothesized parameter value, making the test strongly resemble the Wald test.

You cannot do this calculation without the sample, or sufficient statistics thereof, and the log likelihood function, since the likelihood function needs data to be able to calculate a value! This is how the data gets into the test - through the likelihood function, rather than through an explicit parameter estimate appearing in the formulation of the test itself.

In your case, working through the algebra of $c(\lambda)$ results in a null hypothesis of $\lambda = \sqrt{10}$. I'll toss this out and use the same null hypothesis as with the LR test, as that will be a little more educational.

Let's do a little algebra. We'll start with the derivative of the log likelihood $l = -n\lambda + \sum{x_i}\ln \lambda$:

$\text{d}l/\text{d}\lambda = -n + \sum x_i/\lambda = -n + n\hat{\lambda}/\lambda = n(\hat{\lambda}-\lambda)/\lambda$

Squaring this and dividing by $n/\lambda$ (the Fisher information) gives:

$LM = (\hat{\lambda}-\lambda)^2 / (\lambda/n)$.

If we were to construct a Wald test for this problem, the numerator would be the squared difference between the MLE and the hypothesized value of the parameter $(\hat{\lambda}-\lambda)^2$, and the Fisher information would be evaluated at the MLE $\hat{\lambda}$. Consequently, the test statistic would be:

$W = (\hat{\lambda}-\lambda)^2 / (\hat{\lambda}/n)$.

Substituting in the various values in your example gives the Lagrange Multiplier and Wald test statistics $LM = 4.5$, $W = 5.29$. In both of these cases, we would be able to reject the null hypothesis of $\lambda = 2$ at the 95% but not the 99% level of confidence, as with the Likelihood Ratio test.

Of course, we could have substituted values directly into the initial expression for $\text{d} l/\text{d}\lambda$ and we would have gotten the same answer, but working through the algebra does allow us to observe a not uncommon relationship between the LM and Wald tests, namely, that the numerator is often the same with the denominator only differing in that the LM test uses the null hypothesis parameters and the Wald test uses the MLEs of the parameters.

For these examples, the calculations are pretty simple, because the distributions are simple and we are assuming the observations are independent and identically distributed. If our observations are not independent, typically the calculations become more complex, often far more so, and the number of parameters that need to be estimated can increase substantially too. This also changes a lot of the calculations to matrix calculations instead of scalar calculations, e.g., the Fisher information becomes a matrix instead of a scalar, and expressions of the form $\hat{\lambda} - \lambda$ become vectors rather than scalars, resulting in, for example, $W = (\hat{\theta}-\theta)^TI(\theta)(\hat{\theta}-\theta)$. The expressions also increase in complexity if we consider tests on functions of the parameters rather than directly on the parameter values themselves.

$\endgroup$
  • $\begingroup$ Thank you for your extensive response. Some clarificatory questions before I award you the bounty: 1) Is there a short way to see what the expected Fisher information is for any distribution, e.g. for the poisson or exponential distribution? 2) I don't see how you arrive at 4.74 for the LR test. I calculated a value of -110. Particularly, what do you take as $\bar{x}$ in $l(\theta)$? The value of $\lambda$, right? $\endgroup$ – Taufi Jul 26 '17 at 10:14
  • $\begingroup$ 3) How do you obtain the degrees of freedom for the LM test? And one other thing, when I consider, like above, a random sample from one random variable that is distributed as Poisson, normal, ... - can I then assume that these are independent? This is quite important for the Likelihood function. $\endgroup$ – Taufi Jul 26 '17 at 11:42
  • $\begingroup$ Don't award the bounty before the time is up! You never know if someone will come up with a clearer and more detailed answer. I'll address the other questions by editing the answer later today. $\endgroup$ – jbowman Jul 26 '17 at 15:47
  • $\begingroup$ Thanks for your edits. A question with regard to it: Don't I already consider tests on functions of the parameters with $c(\lambda) = 0$ and the sort? What do you mean with tests on fuctions of the parameters? Also, what would need to happen for the formula to become $W = (\hat{\theta} - \theta)^TI(\theta)(\hat{\theta} - \theta)$, i.e. multidimensional - more simultaneous parameters or just dependence of the observations? In essence, I don't see yet when to use the matrix version. $\endgroup$ – Taufi Jul 26 '17 at 19:03
  • $\begingroup$ 1) A test on functions of the parameters requires more math because you, in effect, have to calculate the information matrix etc. for the functions of the parameters, not for the parameters themselves. None of the math I've presented above contains anything related to tests other than directly on the parameter of interest. An example might be testing that the mean = the variance of a Normal variate; you are testing $\mu - \sigma^2 = 0$. $\endgroup$ – jbowman Jul 26 '17 at 19:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.