13
$\begingroup$

I was under the impression that the function lmer() in the lme4 package didn't produce p-values (see lmer, p-values and all that).

I've been using MCMC generated p values instead as per this question: Significant effect in lme4 mixed model and this question: Can't find p-values in the output from lmer() in the lm4 package in R.

Recently I tried a package called memisc and its getSummary.mer() to get the fixed effects of my model into a csv file. As if by magic, a column called p appears which matches my MCMC p-values extremely closely (and doesn't suffer the processing time that comes with using pvals.fnc()).

I've tentatively had a look at the code in getSummary.mer and have spotted the line that generates the p-value:

p <- (1 - pnorm(abs(smry@coefs[, 3]))) * 2

Does this mean p values can be generated directly from lmer's output rather than running pvals.fnc? I realise this will no doubt start the 'p-value fetishism' debate but I'm interested to know. I've not heard memisc mentioned before when it comes to lmer.

To be more succinct: What is the benefit (if any) of using MCMC p-values over those generated by getSummary.mer()?

Improve this question
$\endgroup$
9
  • 9
    $\begingroup$ I am the original author of the getSummary.mer function. The reported $p$-values should only be used as a quick check. If I recall, I actually only included the $p$-values to make it work within the framework provided by memisc. But this should really be provided with an appropriate warning to the user, and I will contact the package maintainer to see about getting this added. My advice is to follow that provided by Doug Bates: MCMC is the safe bet (assuming others don't have better options). $\endgroup$
    – Jason Morgan
    Commented Sep 3, 2013 at 0:55
  • 4
    $\begingroup$ @JasonMorgan This looks to me to be a pretty reasonable answer to the question. $\endgroup$
    – Glen_b
    Commented Sep 3, 2013 at 1:22
  • 2
    $\begingroup$ @JasonMorgan I agree with what you say but currently mcmcsamp() is not available because of a number of issues (One can check the Status of mcmcsamp section in glmm.wikidot.com/faq for more details). I feel that at the moment time, probably (parametric?) bootstrapping is a viable -and not too hard to implement- alternative; the bootMer() functiom can be of service. $\endgroup$
    – usεr11852
    Commented Sep 3, 2013 at 3:10
  • 1
    $\begingroup$ @JasonMorgan I think a copy-paste would be fine, since to my eyes your valuable comment does actually answer the question. (However, if you feel you can expand even a little on your explanation of why the p-values should really only be used as a quick check rather than as p-values, or when they may be more likely to be poor approximations, or why MCMC is a safe bet, so much the better.) $\endgroup$
    – Glen_b
    Commented Sep 3, 2013 at 6:15
  • 3
    $\begingroup$ One other technical/terminological note to the OP. The p-values obtained using the method in memisc are the p-values from treating the observed test statistics as Wald statistics (treating the t as a Wald z in this case). Such a test relies on the "large sample" assumption and so is more and more trustworthy as your sample sizes grow larger. The MCMC-based value, to my knowledge, does not rely on such an assumption. So anyway, reading a little about Wald tests and alternatives to them could help to shed further light on your question. $\endgroup$
    – Jake Westfall
    Commented Sep 3, 2013 at 6:47

1 Answer 1

Reset to default
11
$\begingroup$

I am the original author of the getSummary.mer function. The reported $p$-values should only be used as a quick check. If I recall, I actually only included the $p$-values to make it work within the framework provided by memisc. But this should really be provided with an appropriate warning to the user, and I will contact the package maintainer to see about getting this added. My advice is to follow that provided by Doug Bates: MCMC is the safe bet (assuming others don't have better options).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.