3
$\begingroup$

I was wondering if I could get some opinions on an issue. I'm analyzing my data using mixed-effects modeling in R (lme4 package). My model has by-subject and by-item intercepts and slopes, and random correlation parameters between them. Since the current version of lmer() does not have MCMC sampling implemented, I cannot get a pvalue for the coefficients in the model. Therefore, I would like to report the t-value instead.

I have often seen papers in my field (psycholinguistics) just say something like "In all models presented, |t| > 2 and |z| > 2 correspond to a significant effect at a significance level of .05". I was wondering whether there is some reference I can provide from this type of sentences? I understand that this tends to be the case, but I wonder whether this is something that has been shown (and I should give references) or whether it is ok to just state it and assume everyone will be ok with it.

Suggestions welcome!

Improve this question
$\endgroup$
5
  • $\begingroup$ The statement for |t|>2 will only be true if the degrees of freedom are large enough. Can you provide examples of papers that make a statement like that? Is it possible for you to use the nlme package instead of lme4? $\endgroup$
    – mark999
    Oct 19, 2013 at 8:24
  • $\begingroup$ Is there a way to present confidence intervals (e.g. of the slopes)? Confidence intervals not encompassing 0 => rejection of nil-null. $\endgroup$
    – jona
    Oct 19, 2013 at 8:44
  • $\begingroup$ @mark999: sure, for example, see [Vasishth, Brüssow, Lewis, Drenhaus, (Cognitive Science, 2008] (ncbi.nlm.nih.gov/pubmed/21635350). They only present t-values in the tables and say below Table 5: "T scores with absolute values greater than 2 are statistically significant." (page 704). $\endgroup$
    – Sol
    Oct 19, 2013 at 15:48
  • $\begingroup$ In Table 5 of that paper, they also give the HPD, and it's obvious that iff the upper and lower edge of the HPD have the same sign, |t| > 2. (In fact, |t| > 1.) .. So basically, report HPDs. $\endgroup$
    – jona
    Oct 19, 2013 at 16:07
  • $\begingroup$ @jona: yes, you are right. The problem is that the functions that would calculate HPD intervals (mcmcsamp, pvals.fnc) are all not implemented in R for models with random correlation parameters. So I can't use those and I don't know how to get them otherwise. Maybe I should ask this as a separate question in case anyone has suggestions? $\endgroup$
    – Sol
    Oct 19, 2013 at 16:28

2 Answers 2

Reset to default
5
$\begingroup$

A reference can be found in footnote 1 of Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59(4), 390–412. I'm quoting the relevant bits here:

For data sets characteristic for studies of memory and language, which typically comprise many hundreds or thousands of observations, the particular value of the number of degrees of freedom is not much of an issue. Whereas the difference between 12 and 15 degrees of freedom may have important consequences for the evaluation of significance associated with a t statistic obtained for a small data set, the difference between 612 and 615 degrees of freedom has no noticeable consequences. For such large numbers of degrees of freedom, the t distribution has converged, for all practical purposes, to the standard normal distribution. For large data sets, significance at the 5% level in a two-tailed test for the fixed effects coefficients can therefore be gauged informally by checking the summary for whether the absolute value of the t-statistic exceeds 2.

As you see, he describes it as "informal".

Generally, I assume you will find many people encourage you to report more informative measures than the probability of the data given an effect of exactly zero; for example, confidence intervals/HPD intervals of standardised effect sizes.

Improve this answer
$\endgroup$
3
  • $\begingroup$ thanks! This is useful. In my study I have ~1500 observations per condition, so I don't know if I would fit Baayen's criteria. I would like to use confidence intervals if I can, but I usually get my confidence intervals from the output of lmer (calculated through MCMC also) and that is again not an option when my mixed-effects model has the random correlation paramenters between intercept and slopes $\endgroup$
    – Sol
    Oct 19, 2013 at 15:34
  • $\begingroup$ @SolLago: I've update the quote, it seems you're in the clear. Do read the paper by Baayen et al. though. $\endgroup$
    – jona
    Oct 19, 2013 at 16:01
  • 1
    $\begingroup$ +1 to your answer jona. As a somewhat general comment: I would question the realistic difference of a $t$-distribution with 600+ d.f. and a Gaussian... Those $t$-values should look awful lot like $z$-values. $\endgroup$
    – usεr11852
    Oct 20, 2013 at 4:50
3
$\begingroup$

Here is an example of the reporting to which you refer.
We took the Baayen article seriously and report only AIC differences and t-values in a relatively large analysis of several variables. We emphasized effect sizes and no p-values appear!

Improve this answer
$\endgroup$
1
  • $\begingroup$ @SolLago, I just edited the answer. Its author is Jeff Long (say thanks to him). :) $\endgroup$
    – Andre Silva
    Oct 19, 2013 at 16:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.