2
$\begingroup$

I am working on a binomial mixed model. I want to analyse the use of a certain construction by students. My response is CONSTRUCTION (two levels: THAT/NO_THAT). My predictors are both categorical (type of INSTRUCTION with five levels (A, B, C, D, E) and L2 with three levels (Italian, Spanish, French). I am using lme4 and sum coding (contr.sum) because I do not want one level of a variable to be the reference level.

I started with a maximal model and used LRTs (anova()) to remove non-significant predictors.

This is my final model:

m1 <- glmer(CONSTRUCTION  ~ 
           L2 * INSTRUCTION
         + (1|ID), 
         data=data2, family="binomial",
         glmerControl(optimizer = "bobyqa", optCtrl = list(maxfun = 100000)))

I am wondering what the p-values given by summary() for the individual regressors exactly mean. I have seen several publications in which these p-values are reported and taken as criteria to decide whether a regressor is significant.

enter image description here

In my case this would mean that regarding the interaction L2 * INSTRUCTION only the term L22:INSTRUCTION2 is significant. However, how do I know if the levels that are not shown in the summary (i.e. e.g. L23:INSTRUCTION1) are significant? Is it really correct to use these p-values to evaluate the significance of individual regressors?

What is the difference between these p-values shown by summary() and e.g. the following output from the package emmeans():

emm_1 <- emmeans(m1, "INSTRUCTION", by = "L2")
pairs(emm_1)

enter image description here

Which p-values should be used to evaluate if the regressors of a model are significant?

Thank you very much!

Edit: Just to make sure that I understand the meaning of the p-values provided by summary() correctly: Does the p-value of a coefficient correspond to a test that tests if the coefficient is significantly different from the grand mean (i.e. for L21, if the first level of L2 (Italian) is significantly different from 0, i.e. the grand mean)?

And how does that meaning of the p-value change if I use treatment coding instead. Then that would be the ouput of summary():

enter image description here

Do significant p-values here indicate that the coefficients are significantly different from the reference level (i.e. for L2Spanish it corresponds to the null-hypothesis that the difference in the log odds of using NO_THAT between Spanish and Italian (in INSTRUCTION A) is equal to 0)? And for the p-value of the interaction L2Spanish:INSTRUCTIONB the null hypothesis is that for students with L2 Spanish there is no difference in the log odds of using NO_THAT between INSTRUCTION A (the reference level) and INSTRUCTION B?

$\endgroup$
1
  • $\begingroup$ The interaction is represented by several terms in the model. You need to perform a nested model test (drop1(m1_t, test="Chi")) to get a test of the interaction. You cannot interpret the individual p-values. I also don't see the point of your aversion to reference level coding here. $\endgroup$ Mar 9, 2023 at 20:18

1 Answer 1

2
$\begingroup$

I think people usually use P values from anova(m1) or car::Anova(m1) which perform omnibus tests on the terms in the model rather than on the individual coefficients. So, my answer would be "neither" - use the anova tests instead to decide on the model.

$\endgroup$
1
  • $\begingroup$ Thank you! I understand that anova() is a good way to decide if terms improve the model. I was just confused by some papers I read which seemed to exclusively rely on the p-values next to the coefficients. I am still a little confused by the meaning of the p-values next to the coefficients. I added a question about the difference in meaning when using treatment coding instead of sum coding. $\endgroup$
    – max22
    Mar 9, 2023 at 19:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.