Skip to main content
added 2391 characters in body
Source Link
AdamO
  • 64.8k
  • 6
  • 134
  • 273

Basically, the Wald statistic is not good and you shouldn't trust it for mixed models. It uses a much cruder approximation to the actual likelihood than you get with the profile and boot.ci methods. If R (and SAS and JMP and...) would have been written today, they would not have bothered implementing Wald stats. That's why the summary.merMod method intentionally omits $p$-values from the fixed effect coefficient output. The computational intensity of profile/bootstrap is at most on the scale of minutes by today's standards, but in the olden days, it would take weeks. So, the analyst was expected to do massive amounts of testing and variable transformation methods so that Wald stat might have good-ish properties.

EDIT: below is a snippet of a conversation between me, David Dahl, and Douglas Bates back in 2010 when I tried to suggest using the Wald $p$-values for xtable.

A user of your lme4 package would like to use xtable on mer objects from lme4. That means defining a function "xtable.mer". He suggests the implementation below. I regrettably am not very familar with lme4. Do you have any suggestions?

I appreciate Adam's suggestion and his providing an implementation. Regrettably, I think that the implementation would be controversial, to say the least, and I would prefer not to be the recipient of the fallout. There is a long-standing issue with lme4 regarding p-values on tests of the fixed-effects parameters. For linear mixed models there is a widespread belief that you can calculate a t-statistic (what is labelled here as a "z value") and convert it to a p-value by the simple expedient of determining an approximate number of degrees of freedom. In fact, SAS PROC MIXED offers several (6, I believe) different, and incompatible, ways of determining such degrees of freedom and the corresponding p-values. The fact that these give different answers doesn't deter people from regarding such approximations as "absolute truth".

In reality the distribution of such a statistic is not a Student's T. It is much more complicated than that and I advocate other ways of calculating confidence intervals or testing hypotheses. In the case of a generalized linear mixed model I do calculate a p-value from the standard normal distribution, not because the approximation is better for GLMMs than for LMMs but because it is worse.

I am writing a book for Springer on lme4 (chapter drafts are available at http://lme4.R-forge.R-project.org/book/) where I describe using likelihood ratio tests for hypothesis tests and techniques based on profiling the LRT statistic to produce confidence intervals on parameters. The examples in that book are based on the development version of the package which uses a different representation of the model. The implementation is not complete, which is why I haven't released it as lme4, but right now I need to concentrate on the writing because the book is going to be used in a seminar which starts next week.

Basically, the Wald statistic is not good and you shouldn't trust it for mixed models. It uses a much cruder approximation to the actual likelihood than you get with the profile and boot.ci methods. If R (and SAS and JMP and...) would have been written today, they would not have bothered implementing Wald stats. That's why the summary.merMod method intentionally omits $p$-values from the fixed effect coefficient output. The computational intensity of profile/bootstrap is at most on the scale of minutes by today's standards, but in the olden days, it would take weeks. So, the analyst was expected to do massive amounts of testing and variable transformation methods so that Wald stat might have good-ish properties.

Basically, the Wald statistic is not good and you shouldn't trust it for mixed models. It uses a much cruder approximation to the actual likelihood than you get with the profile and boot.ci methods. If R (and SAS and JMP and...) would have been written today, they would not have bothered implementing Wald stats. That's why the summary.merMod method intentionally omits $p$-values from the fixed effect coefficient output. The computational intensity of profile/bootstrap is at most on the scale of minutes by today's standards, but in the olden days, it would take weeks. So, the analyst was expected to do massive amounts of testing and variable transformation methods so that Wald stat might have good-ish properties.

EDIT: below is a snippet of a conversation between me, David Dahl, and Douglas Bates back in 2010 when I tried to suggest using the Wald $p$-values for xtable.

A user of your lme4 package would like to use xtable on mer objects from lme4. That means defining a function "xtable.mer". He suggests the implementation below. I regrettably am not very familar with lme4. Do you have any suggestions?

I appreciate Adam's suggestion and his providing an implementation. Regrettably, I think that the implementation would be controversial, to say the least, and I would prefer not to be the recipient of the fallout. There is a long-standing issue with lme4 regarding p-values on tests of the fixed-effects parameters. For linear mixed models there is a widespread belief that you can calculate a t-statistic (what is labelled here as a "z value") and convert it to a p-value by the simple expedient of determining an approximate number of degrees of freedom. In fact, SAS PROC MIXED offers several (6, I believe) different, and incompatible, ways of determining such degrees of freedom and the corresponding p-values. The fact that these give different answers doesn't deter people from regarding such approximations as "absolute truth".

In reality the distribution of such a statistic is not a Student's T. It is much more complicated than that and I advocate other ways of calculating confidence intervals or testing hypotheses. In the case of a generalized linear mixed model I do calculate a p-value from the standard normal distribution, not because the approximation is better for GLMMs than for LMMs but because it is worse.

I am writing a book for Springer on lme4 (chapter drafts are available at http://lme4.R-forge.R-project.org/book/) where I describe using likelihood ratio tests for hypothesis tests and techniques based on profiling the LRT statistic to produce confidence intervals on parameters. The examples in that book are based on the development version of the package which uses a different representation of the model. The implementation is not complete, which is why I haven't released it as lme4, but right now I need to concentrate on the writing because the book is going to be used in a seminar which starts next week.

Source Link
AdamO
  • 64.8k
  • 6
  • 134
  • 273

Basically, the Wald statistic is not good and you shouldn't trust it for mixed models. It uses a much cruder approximation to the actual likelihood than you get with the profile and boot.ci methods. If R (and SAS and JMP and...) would have been written today, they would not have bothered implementing Wald stats. That's why the summary.merMod method intentionally omits $p$-values from the fixed effect coefficient output. The computational intensity of profile/bootstrap is at most on the scale of minutes by today's standards, but in the olden days, it would take weeks. So, the analyst was expected to do massive amounts of testing and variable transformation methods so that Wald stat might have good-ish properties.