9
$\begingroup$

From what I've been reading, amongst others on the site of the UCLA statistics consulting group likelihood ratio tests and wald tests are pretty similar in testing whether two glm models show a significant difference in the fit for a dataset (excuse me if my wording might be a bit off). In essence I can compare two models and test if the second model shows a significantly better fit than the first, or there is no difference between the models.

So the LR and Wald tests should show the same ballpark p-values for the same regression models. At least the same conclusion should come out.

Now I did both tests for the same model in R and get widely differing results. Here are results from R for one model:

> lrtest(glm(data$y~1),glm(data$y~data$site_name,family="poisson"))
Likelihood ratio test

Model 1: data$y ~ 1
    Model 2: data$y ~ data$site_name
      #Df  LogLik Df  Chisq Pr(>Chisq)    
    1   2 -89.808                         
    2   9 -31.625  7 116.37  < 2.2e-16 ***
    ---
    Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
    > lrtest(glm(data$y~1,family="poisson"),glm(data$y~data$site_name,family="poisson"))
Likelihood ratio test

Model 1: data$y ~ 1
    Model 2: data$y ~ data$site_name
      #Df  LogLik Df  Chisq Pr(>Chisq)    
    1   1 -54.959                         
    2   9 -31.625  8 46.667  1.774e-07 ***
    ---
    Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
    > waldtest(glm(data$y~data$site_name,family="poisson"))
Wald test

Model 1: data$y ~ data$site_name
Model 2: data$y ~ 1
      Res.Df Df      F Pr(>F)
    1     45                 
    2     53 -8 0.7398 0.6562
    > waldtest(glm(data$y~1,family="poisson"),glm(data$y~data$site_name,family="poisson"))
Wald test

Model 1: data$y ~ 1
    Model 2: data$y ~ data$site_name
  Res.Df Df      F Pr(>F)
1     53                 
2     45  8 0.7398 0.6562

About the data, data\$y contains count data and data\$site_name is a factor with 9 levels. There are 54 values in data\$y, with 6 values per level of data\$site_name.

Here are frequency distributions:

> table(data$y)

 0  2  4  5  7 
50  1  1  1  1 
> table(data$y,data$site_name)

    Andulay Antulang Basak Dauin Poblacion District 1 Guinsuan Kookoo's Nest Lutoban Pier Lutoban South Malatapay Pier
  0       6        6     6                          4        6             6            6             5              5
  2       0        0     0                          0        0             0            0             1              0
  4       0        0     0                          1        0             0            0             0              0
  5       0        0     0                          0        0             0            0             0              1
  7       0        0     0                          1        0             0            0             0              0

Now this data doesn't fit the poisson distribution very well due to the enormous over-dispersion of zero counts. But with another model, where data\$y>0 fits the poisson model quite well, and while using a zero-inflated poisson model, I still get highly different wald test and lrtest results. There the wald test shows a p-value of 0.03 while the lrtest has a p-value 0.0003. Still a factor 100 difference, even though the conclusion might be the same.

So what am I understanding incorrectly here with the likelihood ratio vs waldtest?

$\endgroup$
14
$\begingroup$

It's important to note that although the likelihood ratio test and the Wald test are used by researchers to accomplish the same empirical goal(s), they are testing different hypotheses. The likelihood ratio test evaluates whether the data were likely to have come from a more complex model, vs. a more simple model. Put another way, does the addition of a particular effect allow the model to account for more information. The Wald test, conversely, evaluates whether it is likely that the estimated effect could be zero. It's a nuanced difference, to be sure, but an important conceptual difference nonetheless.

Agresti (2007) contrasts likelihood ratio testing, Wald testing, and a third method called the "score test" (he hardly elaborates on this test further). From his book (p. 13):

When the sample size is small to moderate, the Wald test is the least reliable of the three tests. We should not trust it for such a small n as in this example (n = 10). Likelihood-ratio inference and score-test based inference are better in terms of actual error probabilities coming close to matching nominal levels. A marked divergence in the values of the three statistics indicates that the distribution of the ML estimator may be far from normality. In that case, small-sample methods are more appropriate than large-sample methods.

Looking at your data and output, it seems that you do indeed have a relatively small sample, and therefore may want to place greater stock in the likelihood ratio test results vs. the Wald test results.

References

Agresti, A. (2007). An introduction to categorical data analysis (2nd edition). Hoboken, NJ: John Wiley & Sons.

$\endgroup$
  • $\begingroup$ The likelihood ratio test evaluates whether the data were likely to have come from a more complex model, vs. a more simple model. Surely it evaluates the slightly different question, "which model would have led to the observed data with the highest probability"? You seem to have described a Bayes Factor or some such, which requires information about the prior probabilities... yes? $\endgroup$ – Jake Westfall Feb 2 '16 at 19:32
  • $\begingroup$ Isn't the difference between the two questions what the second model is you test it against? If you do an lr with y~1 as model1 y~x as model2, then indeed the lr tests whether the data is more probable to be explained by a more complex (model2) or a more simple model (model1). Which is the same in this case as the question "which model is it explained by with the highest probability". Right? $\endgroup$ – Dolf Andringa Feb 2 '16 at 23:03
3
$\begingroup$

First, I disagree somewhat with jsakaluk's answer that the two tests are testing different things - they are both testing whether the coefficient in the larger model is zero. They are just testing this hypothesis by making different approximations (see article linked to below).

Regarding the differences between their results, as jsakaluk said, this is likely due to the small sample size / that the log likelihood is far from quadratic. I wrote a blog post back in 2014 which goes through this for a simple binomial model, which may help further: http://thestatsgeek.com/2014/02/08/wald-vs-likelihood-ratio-test/

$\endgroup$
2
$\begingroup$

The two tests are asymptotically equivalent. Of course, their performance (size and power) in finite samples can differ. The best you can do to understand the difference is to run a Monte Carlo study for a setting similar to yours.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.