The situation:

A, B, C, and D are fixed effects, and RandomEffect is a random effect, all of which are being use to predict the variable Response.

My prediction was that A and B would show a detectable effect on Response, but my analyses showed no such pattern. I assumed C and D to have some effect, but this was based on scientific reasoning (the topic is understudied). As C and D were "control" predictors, they were included in both the full and null model.

I am interested in understanding how to obtain appropriate p-values and confidence intervals for variables which were not part of my prediction (C and D), and how to present my results.

My analysis:

I used a model with all possible random slopes and correlations between random slopes and intercepts (see this paper for more information).

Full model R formula: (A, B, C, and D impact Response)

full = Response ~ A + B + C + D + (1 + A + B + C + D | RandomEffect)

Null model R formula: (only C, and D impact Response)

null = Response ~ C + D + (1 + A + B + C + D | RandomEffect)


The full-reduced model comparison - anova(null, full, test="Chisq") - was not significant, i.e. no evidence A and B impact Response. However, C and D do have a detectable effect. Since there is not much known about C and D’s effect on Response, it would be interesting to discuss it in my paper.

My questions are...

  1. Even though my full-null model comparison was not significant, is it still acceptable to report the coefficients / p-values / confidence intervals / etc. for C and D?

  2. If yes, would I use values derived from...

    • a. The full model
    • b. The null model
    • c. A simpler model excluding all aspects of the main effects, like:
      • Response ~ C + D + (1 + C + D | RandomEffect)
  3. If the answer to 2 is b. or c., then would I further test the null/simplified model against another, further simplified, model which excludes C and D? Would I then need to make a correction for multiple testing?

  • $\begingroup$ You state at the beginning that your goal is to test the effects of A and B, but your questions are all based on analyses that say that A & B have no detectable effect. Could you clarify what you are trying to achieve? $\endgroup$
    – mkt
    Jul 4, 2017 at 10:40
  • $\begingroup$ My prediction was that A and B have an effect on Response. My data show that they do not. My assumption was that C and D have an effect on Response, and the data show they do. My questions are about how to report the significance/effect of C and D on Reponse, even though they are not part of my original prediction. Does that clarify the question? Is there something else that needs more explanation? Please let me know and I will try to clarify more. Thanks! $\endgroup$
    – Robyn
    Jul 4, 2017 at 11:27
  • $\begingroup$ Thanks, that helps (and I have submitted edits to clarify your question). $\endgroup$
    – mkt
    Jul 4, 2017 at 11:41

1 Answer 1


1) You can report the coefficients, p-values, confidence intervals etc whether significant or not.

2) I would favour using co-efficients from the full model, as you have clear a priori reason to include all the parameters that you did. However, it is not uncommon to present co-efficient estimates from a reduced model with only the significant terms retained.

You can obtain p-values for each term in multiple ways, but one of the most reliable methods for mixed models is a bootstrap-based approach. See Halekoh & Højsgaard 2014 (https://www.jstatsoft.org/article/view/v059i09) and their associated R package 'pbkrtest', which makes this easy to do. To briefly describe the approach, you construct a series of nested models with individual terms dropped, and you perform a bootstrap-based comparison between each reduced model and the full model to evaluate whether the dropped terms contribute to overall explanatory power. You can then construct a final model with only the significant terms (and the random effect) retained, if you wish.

3) I think the answer to #2 covers this.

Let me know if this addresses your question.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.