8
$\begingroup$

I've run into this issue a few times now, with reviewers requesting more justification for the use of LMMs, traditional tests instead of or in addition to LMMs, and full tables of parameter estimates akin to what you would report with a regular linear model.

Right now my specific issue is a reviewer requesting "A table containing the main parameter estimates of the various models". I am thinking they want something like a traditional table one would report for a linear model (with t tests and p values), but in this case the analyses involve nested model comparisons and there are no t tests for each of the parameters included in each model, but rather a single test for the model comparison, which I do report in the paper. So I'm not sure what to do -- I want to satisfy the reviewer, but I don't necessarily want to include huge tables of information that are of little use to evaluating the results. Right now I simply report the beta, SE, chi-square and p value. I also make it clear what variables were included in each model. Any suggestions for how to proceed?

Here is what I am proposing to respond:

We believe the reviewer is asking for something akin to what would be reported in a traditional multiple regression analysis, with parameter estimates and their accompanying statistics and p values for each variable included in a given model. However, because linear mixed model analyses use nested models comparing reduced models to full models with one additional parameter, the only parameter that is tested is the one that is added in the full model (CITATION) As such, including a table would not support interpretation of the results in the way that it would in a more traditional analysis. Thus for each analysis, we report the betas for the tested parameter in each model comparison, along with the key statistics, in the body of the results section, as is recommended (CITATIONS).

Also, when asked for a justification for the use of LMMs in my particular case, this is what I'm proposing to respond:

We used linear mixed models because this analysis allowed us to account for variability due to trial type in our models (switch versus no-switch trials), while simultaneously accounting for the fact that trials were nested within subjects, and multiple responses from the same person are more similar than responses from other people. Accounting for both trial type and subject-level variance in reaction times was expected to reduce error in our models and increase our ability to detect any effect of task performance.

If you have any suggestions for how this could be improved, I'd appreciate it. Again, this audience is not statistically sophisticated, so adding tables and supplementary data is only likely to add to their confusion/skepticism.

Also, note that my motivation for using LMMs is different from what I've seen in papers (e.g., modeling multiple random effects simultaneously - in my case, there's only one random effect - participants, and trial type is a fixed effect), so I am not sure that citing some of the common papers is that helpful. It's possible that I've overlooked other ways to analyze this data so my justification for using LMMs is not apt.

$\endgroup$
  • $\begingroup$ Can you give some more information about the model that you use? How complicated is it? What are "various models" that the reviewer is talking about? Do you discuss nested models in your manuscript? $\endgroup$ – amoeba says Reinstate Monica Feb 3 '17 at 12:03
  • $\begingroup$ I report several model comparisons involving a reduced model with some key covariates and a full model with the same variables plus the independent variable of interest. In the analytic approach section of my manuscript I explain exactly how this all will work, but the reviewers clearly aren't familiar with the approach so still have this expectation that i think is guided by their familiarity with multiple regression. $\endgroup$ – panpsych77 Feb 3 '17 at 15:52
  • 2
    $\begingroup$ I don't really see the problem. You can just report all the parameter estimates and SEs for both models, either in two separate tables or combined into one table, and then note: "a likelihood ratio test comparing these two models yielded..." Alternatively, you could do the exhaustive set of model comparisons and report the LRT statistics next to each fixed parameter estimate in the full model table, using e.g. drop1(merMod, type="chisq") $\endgroup$ – Jake Westfall Feb 4 '17 at 21:49
  • $\begingroup$ @JakeWestfall, thanks! I was just under the impression that the convention was not to report all of this info (based on psych papers I've read reporting LMMs), but now I can see why it would make sense to do so. Do you have any examples from your own papers or elsewhere of how you'd format these tables? Similar to regular multiple regression tables I suppose? I can come up with a way that seems intuitive to me but always good to have examples. $\endgroup$ – panpsych77 Feb 5 '17 at 1:34
  • 2
    $\begingroup$ @panpsych77 I routinely request this info while reviewing, mainly so that it's absolutely clear to readers what the full model specification was. Here are a couple examples of how we've formatted such tables in papers I've coauthored: jakewestfall.org/publications/ANES_supplement.pdf jakewestfall.org/publications/femininity.pdf $\endgroup$ – Jake Westfall Feb 5 '17 at 2:07
7
$\begingroup$

I partly take side with the reviewer on this one. You are interested in the effect of your parameter of interest — given the rest of the model. It is hard to interpret the results and to check the the validity of the model if you only report a single parameter of interest. I would provide:

  • the formula of your model
  • beta estimates for all fixed effects
  • corresponding SEs and CIs
  • corresponding test statistics (z, t, Chi^2, change in AIC/BIC, whatever you used) with df's/n's
  • corresponding p values
  • SDs for your random effects and their correlations (if necessary as separate table)

The space constraints in most classical journals will make it necessary to put these information into an online supplement.

Examples for reporting mixed models can be found here.

$\endgroup$
  • $\begingroup$ I've been looking for guidelines on reporting mixed models in biomedical/psychological research for some time now, but haven't found any. I'd be happy for any citeable references. $\endgroup$ – mzunhammer Feb 3 '17 at 16:30
  • $\begingroup$ The only guidelines I've found are online tutorials, which don't seem appropriate. I might just cite the R book (Crawley) or some psych papers that report LMMs insofar as they establish a precedent. $\endgroup$ – panpsych77 Feb 3 '17 at 17:22
  • $\begingroup$ Also, could you read what I added to my original post above and let me know if it makes sense? Thank you! $\endgroup$ – panpsych77 Feb 3 '17 at 17:22
  • 2
    $\begingroup$ Updated the answer above. Another point you could use to justify the use of LMM analysis rather than standard LM analysis is that it is necessary to account for repeated measures, since failure to do so would violate the assumption of independent observations. $\endgroup$ – mzunhammer Feb 4 '17 at 12:37
  • 2
    $\begingroup$ Readers will often want to see results presented in a way that makes it easy to compare a new study against previous studies. For example, in survival analysis, journals often prefer to show large tables of many single-variable relations to outcome even though those relations are of limited value statistically. Such tables provide some reassurance that the new patient cohort is similar to other cohorts. I agree to provide more rather than less in terms of results tables; explanation of the limits of the tables then serves a useful educational function for the readers. $\endgroup$ – EdM Feb 4 '17 at 15:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.