Citation for ML vs. REML

Question

Quick question: can anyone give me a citation that I can use to justify using ML when doing model comparisons?

Background: I am fitting some multilevel models in R using lme4, and I do a series of model comparisons. One reviewer told me I should only use REML (never ML) when running a model or when comparing models. I have since heard quite the opposite with regard to model comparisons, and I was told elsewhere that ML is fine. The models are not complex.... one or two level-2 predictors with no level-1 predictors. Nice and simple.

I would love (1) confirmation that I can validly run a model without REML [ever] and a citation for that, and (2) some form of citation that says you need ML (NOT REML) when doing LRT comparisons (BIC and AIC also support me, but I wanted a p-value to accompany them).

Any advice you give, if you have citations, that would be great as my reviewers want them.

Best,

Answer 1

Sigh.

• Comparing models that are fitted with REML and differ in their fixed effects never makes sense.
• Using AIC/BIC/p-values to compare the same model fitted with REML vs ML never makes sense; you need to make the decision which method to use on a priori, theoretical grounds.
• Comparing models that are fitted with REML and differ in their random effects is justified.

This is mentioned (but only in passing) in

Bates D, Maechler M, Bolker BM and Walker S (2014). “lme4: Linear mixed-effects models using Eigen and S4.” ArXiv e-print; in press, Journal of Statistical Software: http://arxiv.org/abs/1406.5823

For objects of class lmerMod the default behavior is to refit the models with ML if fitted with REML = TRUE, which is necessary in order to get sensible answers when comparing models that differ in their fixed effects ...

And more thoroughly, Pinheiro and Bates Mixed-Effects Models in S and S-PLUS 2000 (Springer) p. 87 (scraps from Google Books):

When two nested models differ in the specification of their fixed-effects terms, a likelihood ratio test can be defined for maximum likelihood fits only. As described in section 2.2.5 a likelihood ratio test for REML fits is not feasible, because there is a term in the REML criterion that changes with the change in the fixed-effects specification.

... they then go on to point out that LRT is asymptotic, and finite-size effects (the dreaded "denominator degrees of freedom" problem) can make such tests anticonservative.

This question gives a quotation from Discovering Statistics Using SPSS 4e that also backs up this statement (i.e. it's not just R-using weirdos).