7
$\begingroup$

I have some problems with respect to the degrees of freedom of my lmer model.

This is the output:

lmF3<-lmer(var ~ cond  + (1|subj) + (1|blocks) , data=data)


                Estimate Std. Error     df     t value Pr(>|t|)    
(Intercept)     -0.12844    0.04238    2.30000  -3.031   0.0794 .  
B                0.28143    0.04584 1002.20000   6.139 1.19e-09 ***

I've an unbalanced design, a total 1000 obs, 200 in condition A, 800 in condition B.

I've some doubts about the df of the model... in particular about the df of the intercept (2.3), but even of the constrast A vs B (1002.2).

I think there are some problems... but I dunno precisely what could be. Are those df plausible, or do you think something went wrong?

$\endgroup$
9
  • 2
    $\begingroup$ What is the question? $\endgroup$
    – Firebug
    Commented Apr 19, 2017 at 13:08
  • 1
    $\begingroup$ Please show the command(s) you gave to lmer (and possibly also to lmerTest; as I recall, lmer does not itself report degrees of freedom). Note that the df reported are approximations; see for example this R-help thread. $\endgroup$
    – EdM
    Commented Apr 19, 2017 at 14:20
  • 1
    $\begingroup$ I edited the post, and I wrote the command. Thanks $\endgroup$
    – piravi
    Commented Apr 19, 2017 at 14:28
  • 1
    $\begingroup$ Can you specify why these df values are not plausible? $\endgroup$
    – amoeba
    Commented Apr 20, 2017 at 10:38
  • 3
    $\begingroup$ Note that for t-value of 6.1, you really don't care about dfs. It's very significant for any number of degrees of freedom. $\endgroup$
    – amoeba
    Commented Apr 20, 2017 at 10:43

2 Answers 2

6
$\begingroup$

Are you using the lmerTest package to get your p-values to output for the summary of an lmer object? If so, you are estimating degrees of freedom using the Satterthwaite approximation. The top of your output should say something like:

t-tests use  Satterthwaite approximations to degrees of freedom

I assume you are asking if these dfs are plausible because they are not integers—they have decimals? Since there is no straightforward way of calculating dfs for these multilevel models, different approximations are used, which gives you the dfs that aren't necessarily integers.

$\endgroup$
4
$\begingroup$

How do you define "plausible" degrees of freedom?

Simply speaking, calculating degrees of freedom for GLMMs is complicated and there is no simple formula for calculating them. Let me quote the r-sig-mixed-models FAQ:

(There is an R FAQ entry on this topic, which links to a mailing list post by Doug Bates (there is also a voluminous mailing list thread reproduced on the R wiki). The bottom line is

  • In general it is not clear that the null distribution of the computed ratio of sums of squares is really an F distribution, for any choice of denominator degrees of freedom. While this is true for special cases that correspond to classical experimental designs (nested, split-plot, randomized block, etc.), it is apparently not true for more complex designs (unbalanced, GLMMs, temporal or spatial correlation, etc.).
  • For each simple degrees-of-freedom recipe that has been suggested (trace of the hat matrix, etc.) there seems to be at least one fairly simple counterexample where the recipe fails badly.
  • Other df approximation schemes that have been suggested (Satterthwaite, Kenward-Roger, etc.) would apparently be fairly hard to implement in lme4/nlme, both because of a difference in notational framework and because naive approaches would be computationally difficult in the case of large data sets. (The Kenward-Roger approach has now been implemented in the pbkrtest package (as KRmodcomp): although it was derived for LMMs, Stroup [29] states on the basis of (unpresented) simulation results that it actually works reasonably well for GLMMs. However, at present the code in KRmodcomp only handles LMMs.)
  • Note that there are several different issues at play in finite-size (small-sample) adjustments, which apply slightly differently to LMMs and GLMMs.
    • When the responses are normally distributed and the design is balanced, nested etc. (i.e. the classical LMM situation), the scaled deviances and differences in deviances are exactly F-distributed and looking at the experimental design (i.e., which treatments vary/are replicated at which levels) tells us what the relevant degrees of freedom are.
    • When the data are not classical (crossed, unbalanced, R-side effects), we might still guess that the deviances etc. are approximately F-distributed but that we don't know the real degrees of freedom — this is what the Satterthwaite, Kenward-Roger, Fai-Cornelius, etc. approximations are supposed to do.
    • When the responses are not normally distributed (as in GLMs and GLMMs), and when the scale parameter is not estimated (as in standard Poisson- and binomial-response models), then the deviance differences are only asymptotically F- or chi-square-distributed (i.e. not for our real, finite-size samples). In standard GLM practice, we usually ignore this problem; there is some literature on finite-size corrections for GLMs under the rubrics of "Bartlett corrections" and "higher order asymptotics" (see McCullagh and Nelder, work by Cordeiro, and the cond package on CRAN [which works with GLMs, not GLMMs]), but it's rarely used. (The bias correction/Firth approach implemented in the brglm package attempts to address the problem of finite-size bias, not finite-size non-chi-squaredness of the deviance differences.)
    • When the scale parameter in a GLM is estimated rather than fixed (as in Gamma or quasi-likelihood models), it is sometimes recommended to use an F test to account for the uncertainty of the scale parameter (e.g. Venables and Ripley recommend anova(…,test="F") for quasi-likelihood models)
    • Combining these issues, one has to look pretty hard for information on small-sample or finite-size corrections for GLMMs: Feng et al 2004 [14] and Bell and Grunwald 2010 [6] look like good starting points, but it's not at all trivial.
  • Because the primary authors of lme4 are not convinced of the utility of the general approach of testing with reference to an approximate null distribution, and because of the overhead of anyone else digging into the code to enable the relevant functionality (as a patch or an add-on), this situation is unlikely to change in the future.
$\endgroup$
2
  • $\begingroup$ The fact is that, I have conducted various indipendent analysis using the same sample and the same unbalanced design and I found that the df change a lot: sometimes the df are identical for the intercept and for the contrast A vs B, sometimes they substantially differ, like in the example I reported. As I have to write type the results of the experiment, I wonder whether it could be the cues of a potential error $\endgroup$
    – piravi
    Commented Apr 19, 2017 at 14:50
  • 3
    $\begingroup$ @piravi as stated above, df's for GLMMs are not "reliable" and the general rationale of lme4 authors was not to output them at all because of that. lmer prints them to provide backward compatibility, but this does not make them any better. df's and p-values for GLMMs are complicated and there is no out-of-the box solution for them. If you really need them, then make deeper research on this problem and calculate the values by hand in a way that fitts your specific problem. $\endgroup$
    – Tim
    Commented Apr 19, 2017 at 14:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.