Significance of the control predictors in a non-significant full-null model comparison (mixed model)

Question

The situation:

A, B, C, and D are fixed effects, and RandomEffect is a random effect, all of which are being use to predict the variable Response.

My prediction was that A and B would show a detectable effect on Response, but my analyses showed no such pattern. I assumed C and D to have some effect, but this was based on scientific reasoning (the topic is understudied). As C and D were "control" predictors, they were included in both the full and null model.

I am interested in understanding how to obtain appropriate p-values and confidence intervals for variables which were not part of my prediction (C and D), and how to present my results.

My analysis:

I used a model with all possible random slopes and correlations between random slopes and intercepts (see this paper for more information).

Full model R formula: (A, B, C, and D impact Response)

full = Response ~ A + B + C + D + (1 + A + B + C + D | RandomEffect)

Null model R formula: (only C, and D impact Response)

null = Response ~ C + D + (1 + A + B + C + D | RandomEffect)

Results:

The full-reduced model comparison - anova(null, full, test="Chisq") - was not significant, i.e. no evidence A and B impact Response. However, C and D do have a detectable effect. Since there is not much known about C and D’s effect on Response, it would be interesting to discuss it in my paper.

My questions are...

1. Even though my full-null model comparison was not significant, is it still acceptable to report the coefficients / p-values / confidence intervals / etc. for C and D?

2. If yes, would I use values derived from...

   • a. The full model
   • b. The null model
   • c. A simpler model excluding all aspects of the main effects, like:
     • Response ~ C + D + (1 + C + D | RandomEffect)

3. If the answer to 2 is b. or c., then would I further test the null/simplified model against another, further simplified, model which excludes C and D? Would I then need to make a correction for multiple testing?

Answer 1

1) You can report the coefficients, p-values, confidence intervals etc whether significant or not.

2) I would favour using co-efficients from the full model, as you have clear a priori reason to include all the parameters that you did. However, it is not uncommon to present co-efficient estimates from a reduced model with only the significant terms retained.

You can obtain p-values for each term in multiple ways, but one of the most reliable methods for mixed models is a bootstrap-based approach. See Halekoh & Højsgaard 2014 (https://www.jstatsoft.org/article/view/v059i09) and their associated R package 'pbkrtest', which makes this easy to do. To briefly describe the approach, you construct a series of nested models with individual terms dropped, and you perform a bootstrap-based comparison between each reduced model and the full model to evaluate whether the dropped terms contribute to overall explanatory power. You can then construct a final model with only the significant terms (and the random effect) retained, if you wish.

3) I think the answer to #2 covers this.

Let me know if this addresses your question.