Degrees freedom reported by the lmer model don't seem plausible

Question

I have some problems with respect to the degrees of freedom of my lmer model.

This is the output:

lmF3<-lmer(var ~ cond  + (1|subj) + (1|blocks) , data=data)


                Estimate Std. Error     df     t value Pr(>|t|)    
(Intercept)     -0.12844    0.04238    2.30000  -3.031   0.0794 .  
B                0.28143    0.04584 1002.20000   6.139 1.19e-09 ***

I've an unbalanced design, a total 1000 obs, 200 in condition A, 800 in condition B.

I've some doubts about the df of the model... in particular about the df of the intercept (2.3), but even of the constrast A vs B (1002.2).

I think there are some problems... but I dunno precisely what could be. Are those df plausible, or do you think something went wrong?

Answer 1

Are you using the lmerTest package to get your p-values to output for the summary of an lmer object? If so, you are estimating degrees of freedom using the Satterthwaite approximation. The top of your output should say something like:

t-tests use  Satterthwaite approximations to degrees of freedom

I assume you are asking if these dfs are plausible because they are not integers—they have decimals? Since there is no straightforward way of calculating dfs for these multilevel models, different approximations are used, which gives you the dfs that aren't necessarily integers.

Answer 2

How do you define "plausible" degrees of freedom?

Simply speaking, calculating degrees of freedom for GLMMs is complicated and there is no simple formula for calculating them. Let me quote the r-sig-mixed-models FAQ:

(There is an R FAQ entry on this topic, which links to a mailing list post by Doug Bates (there is also a voluminous mailing list thread reproduced on the R wiki). The bottom line is

• In general it is not clear that the null distribution of the computed ratio of sums of squares is really an F distribution, for any choice of denominator degrees of freedom. While this is true for special cases that correspond to classical experimental designs (nested, split-plot, randomized block, etc.), it is apparently not true for more complex designs (unbalanced, GLMMs, temporal or spatial correlation, etc.).
• For each simple degrees-of-freedom recipe that has been suggested (trace of the hat matrix, etc.) there seems to be at least one fairly simple counterexample where the recipe fails badly.
• Other df approximation schemes that have been suggested (Satterthwaite, Kenward-Roger, etc.) would apparently be fairly hard to implement in lme4/nlme, both because of a difference in notational framework and because naive approaches would be computationally difficult in the case of large data sets. (The Kenward-Roger approach has now been implemented in the pbkrtest package (as KRmodcomp): although it was derived for LMMs, Stroup [29] states on the basis of (unpresented) simulation results that it actually works reasonably well for GLMMs. However, at present the code in KRmodcomp only handles LMMs.)
• Note that there are several different issues at play in finite-size (small-sample) adjustments, which apply slightly differently to LMMs and GLMMs.
  • When the responses are normally distributed and the design is balanced, nested etc. (i.e. the classical LMM situation), the scaled deviances and differences in deviances are exactly F-distributed and looking at the experimental design (i.e., which treatments vary/are replicated at which levels) tells us what the relevant degrees of freedom are.
  • When the data are not classical (crossed, unbalanced, R-side effects), we might still guess that the deviances etc. are approximately F-distributed but that we don't know the real degrees of freedom — this is what the Satterthwaite, Kenward-Roger, Fai-Cornelius, etc. approximations are supposed to do.
  • When the responses are not normally distributed (as in GLMs and GLMMs), and when the scale parameter is not estimated (as in standard Poisson- and binomial-response models), then the deviance differences are only asymptotically F- or chi-square-distributed (i.e. not for our real, finite-size samples). In standard GLM practice, we usually ignore this problem; there is some literature on finite-size corrections for GLMs under the rubrics of "Bartlett corrections" and "higher order asymptotics" (see McCullagh and Nelder, work by Cordeiro, and the cond package on CRAN [which works with GLMs, not GLMMs]), but it's rarely used. (The bias correction/Firth approach implemented in the brglm package attempts to address the problem of finite-size bias, not finite-size non-chi-squaredness of the deviance differences.)
  • When the scale parameter in a GLM is estimated rather than fixed (as in Gamma or quasi-likelihood models), it is sometimes recommended to use an F test to account for the uncertainty of the scale parameter (e.g. Venables and Ripley recommend anova(…,test="F") for quasi-likelihood models)
  • Combining these issues, one has to look pretty hard for information on small-sample or finite-size corrections for GLMMs: Feng et al 2004 [14] and Bell and Grunwald 2010 [6] look like good starting points, but it's not at all trivial.
• Because the primary authors of lme4 are not convinced of the utility of the general approach of testing with reference to an approximate null distribution, and because of the overhead of anyone else digging into the code to enable the relevant functionality (as a patch or an add-on), this situation is unlikely to change in the future.