7
$\begingroup$

I understand the problem of determining the degrees of freedom in multi-level models; hence, the decision by Doug Bates et al. not to report p values as part of the lme4 package in R. Not to mention the plethora of problems with, and undue focus on p values, in general.

However, I would like to clarify the nature of the "t value" reported in the summary output of a multi-level model in nlme or lme4.

Isn't it the case that the reported t value in nlme/lme4 from a data set comprising correlated data is actually not from the t-distribution? (regardless of whether we know the degrees of freedom or not).

Isn't the "t value" in lme4 potentially misleading.

$\endgroup$
9
$\begingroup$

Basically $t$ is just $\beta/\mathrm{SE}(\beta)$, where $\beta$ is regression parameter. There is nothing misleading in this value if you consider it as this ratio, or as "standarized" parameter. If you look at Bates' original arguments against $p$-values in lme4 he writes mostly about the degrees of freedom that are problematic rather than the $t$ of $F$ values themselves (see also r-sig-mixed-models FAQ). Notice that different statistical software can have different naming convention, e.g. as SPSS calls parameters as $B$'s and standarized parameters as $\beta$'s -- lme4 follows the lm convention to call them Estimate and t value.

Pinheiro and Bates describe usage of $p$-values in "Mixed-Effects Models in S and S-PLUS", so it is hard to look for arguments against them in this book. The ratios are also discussed by Bates in "lme4: Mixed-effects modeling with R" in comparison to $t$ and $F$ values for fixed effects models, for example (p. 70):

In a fixed-effects model the profile traces in the original scale will always be straight lines. For mixed models these traces can fail to be linear, as we see here, contradicting the widely-held belief that inferences for the fixed-effects parameters in linear mixed models, based on $T$ or $F$ distributions with suitably adjusted degrees of freedom, will be completely accurate. The actual patterns of deviance contours are more complex than that.

what makes them somehow similar while not exactly adequate as we would expect them to be for proper hypothesis testing.

Notice also that other authors not always consider the df issue to be problematic, e.g. Gałecki and Burzykowski in "Linear Mixed-Effects Models Using R" just assume $n-p$ degrees of freedom and treat their distribution as approximately $t$, e.g. (p. 84):

The null distribution of the $t$-test statistic is the $t$-distribution with $n − p$ degrees of freedom.

and (p. 140):

Confidence intervals for individual components of the parameter vector $\beta$ can be constructed based on a $t$-distribution used as an approximate distribution for the test statistic

So it seems that the main rationale is that while $p$-values can be misleading because of unclear null distribution, $t$ values can still be useful, at least as standardized parameters. You can also use them for hypothesis testing but you need to make some assumption about their distribution and verify them by looking at profile plots. What Bates seems to be saying is that you use them at your own risk.

$\endgroup$
5
$\begingroup$

Correct, the Wald statistic (reported as a "t statistic" by lme4) is, in general, at best only approximately t-distributed for linear mixed models (LMMs). It is only exactly t-distributed in certain very special cases, for example, mixed-model ANOVA with nested random factors and balanced data.

For generalized linear mixed models (GLMMs) with a non-normal response, the distribution of the Wald statistic might not even be very t-like at all. For example, see this thread on logistic regression, where we show that the tails of the sampling distribution can tend be be thinner-than-normal rather than thicker-than-normal. (That thread does not focus on mixed models, but the same issue arises there.)

$\endgroup$
  • $\begingroup$ Thanks for the response Jake. I'm interested in your comment that the lmd t-value will be exactly t-distributed in a nested, balanced, mixed model ANOVA. If our data are not independent won't the distribution of the sample variance be affected by the intraclass correlation (ICC). i.e. the sample variance won't come from a standard Ch-squared; hence, the associated t-statistic won't come from a standard t-distribution. Wont it need to be multiplied by a coefficient that includes the ICC. $\endgroup$ – Rob Casson Dec 7 '15 at 20:58
  • $\begingroup$ @RobCasson Wouldn't your argument lead us to conclude that the t-statistics from any design involving random effects (including, e.g., a simple pre-test vs. post-test design analyzed with a paired-samples t-test) do not follow a t-distribution? Which is not the case. You might try consulting an ANOVA textbook, such as those by Winer or Kirk, where the derivations are laid out for the special cases that I mentioned. $\endgroup$ – Jake Westfall Dec 9 '15 at 17:10
  • $\begingroup$ For pre and post test designs using a paired t-test, there's no problem because the differences are independent, and their variance comes from a routine Chi-squared, and provided these differences are normally distributed then the t-distribution follows. I'm thinking of a multi-level (hierarchical) model, where it's not obvious that, even if the design is balanced that the quotient of the regression parameter estimate and its standard error comes from a t-distribution. For a mixed model ANOVA, I'm confident you are right. $\endgroup$ – Rob Casson Dec 11 '15 at 1:29
  • $\begingroup$ A multi-level model with balanced data and categorical predictors IS a type of mixed model ANOVA. Such models are covered in ANOVA textbooks like the ones I recommended. So if you accept that it's true for mixed model ANOVA then you must accept that it's also true for your non-obvious case. $\endgroup$ – Jake Westfall Dec 11 '15 at 1:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.