I am working on a project looking at ant recruitment to bait in the lab. I want to see which baits ('treatment') the ants recruit to more/less and have multiple measurements for each colony over the course of multiple weeks. I have (hopefully) accounted for varience between colonies and each measurement collection by nesting 'colony' in 'DAT' (days after treatment). My data was overdispersed so I added an observation level random factor to correct for this ('obs'). I am using the lme4 package for the overall glmer model with results below


> mod2015<-glmer(numants~Treatment+(1|colony)+(1|DAT)+(1|obs), family=poisson, data=ncl2015)
> summary(mod2015)

Generalized linear mixed model fit by maximum
  likelihood (Laplace Approximation) [glmerMod]
 Family: poisson  ( log )
numants ~ Treatment + (1 | colony) + (1 | DAT) + (1 | obs)
   Data: ncl2015

     AIC      BIC   logLik deviance df.resid 
  1281.8   1302.6   -633.9   1267.8      137 

Scaled residuals: 
     Min       1Q   Median       3Q      Max 
-2.82084 -0.26069  0.02638  0.15502  0.49829 

Random effects:
 Groups Name        Variance Std.Dev.
 obs    (Intercept) 0.1954   0.4420  
 colony (Intercept) 0.1519   0.3897  
 DAT    (Intercept) 0.1275   0.3571  
Number of obs: 144, groups:  
obs, 144; colony, 24; DAT, 6

Fixed effects:
             Estimate Std. Error z value Pr(>|z|)    
(Intercept)    3.1883     0.2310  13.801   <2e-16 ***
TreatmentHON   0.5283     0.2522   2.095   0.0362 *  
TreatmentKB    0.5512     0.2522   2.186   0.0288 *  
TreatmentPB    0.1277     0.2528   0.505   0.6136    
Signif. codes:  
0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Correlation of Fixed Effects:
            (Intr) TrtHON TrtmKB
TreatmntHON -0.551              
TreatmentKB -0.551  0.504       
TreatmentPB -0.549  0.503  0.503

and am using the multcomp package to conduct pairwise comparisons using the glht function with the results below


> mcpncl2015<-glht(mod2015, mcp(Treatment="Tukey"))
> summary(mcpncl2015)

     Simultaneous Tests for General Linear Hypotheses

Multiple Comparisons of Means: Tukey Contrasts

Fit: glmer(formula = numants ~ Treatment + (1 | colony) + (1 | DAT) + 
    (1 | obs), data = ncl2015, family = poisson)

Linear Hypotheses:
                Estimate Std. Error z value Pr(>|z|)
HON - BUFF == 0  0.52830    0.25223   2.095    0.155
KB - BUFF == 0   0.55117    0.25218   2.186    0.127
PB - BUFF == 0   0.12769    0.25285   0.505    0.958
KB - HON == 0    0.02287    0.25109   0.091    1.000
PB - HON == 0   -0.40061    0.25179  -1.591    0.384
PB - KB == 0    -0.42348    0.25173  -1.682    0.333
(Adjusted p values reported -- single-step method)

PROBLEM As you can see, I get significant differences (p values) when from the over all glmer model summary but those significant differences are not detected when I run the glht function for multiple comparisons using Tukey post hoc test. Can anyone tell me why and how to remedy this? I hope I've explained the issue well enough for help.

  • $\begingroup$ The Tukey test is more stringent than the glmer test. $\endgroup$ Nov 15 '16 at 2:39
  • $\begingroup$ @DJohnson Glmer is not a test. $\endgroup$ Nov 15 '16 at 15:37
  • 1
    $\begingroup$ @subhashc.davar Thanks. Of course that's correct. I should have said than the test results produced by glmer. $\endgroup$ Nov 15 '16 at 15:43
  • $\begingroup$ I'm sorry for the delay in commenting. I had issues with merging 2 accounts. I am by no means a statistician so please forgive my ignorance. I was under the impression that the results from glmer (pvalues) show which level of the fixed factor significantly differs from the intercept (in this case, the level BUFF) based on the model. Is this incorrect? If my interpretation IS correct, why does the significant differences not also show in the pairwise comparisons? $\endgroup$ Nov 21 '16 at 20:54
  • $\begingroup$ @subhashc. Am I correct or incorrect in my interpretation in the above comment? and why would the model indicate significant pvalues (in comparison with the BUFF Treatment as the intercept) while the post hoc test did not? $\endgroup$ Nov 23 '16 at 20:49

BOTH glmer and mcp (Tukey) results show that the treatment HON and treatment KB have a significant effect according to the z-values. The two methods glmer and mcp Tukey method test for significance in different ways. Your interpretation is not correct with regard to mcp (Tukey).

  • $\begingroup$ Thank you for your answer. To clarify, I should ignore the adjusted pvalue of the Tukey test and focus on the z value which shows significant differences between HON-BUFF and KB-BUFF in the results for both mcp and glmer, correct? Is the high Pr(>|z|) from the mcp a result of lowering the critical value for the individual pair comparisons in order to maintain an overall critical value of 0.05 for all comparosons/for the overall model? $\endgroup$ Nov 29 '16 at 1:25

The results are quite consistent.

If you looked at each variable in isolation and had pre-specified that one variable to look at and had pre-specified that all the other variables are in the model, then you can in a null-hypothesis testing framework directly use the p-values from glmer and e.g. HON would only have a coefficient this significant under the null hypothesis (and assuming your model is right) 3.6% of the time.

If you are looking at multiple variables, then you have a multiple comparisons problems and you would expect to get some (i.e. for at least one of the many variables you look at) unadjusted p-value this low over 5% of the time under the null hypothesis.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.