Pairwise comparisons of contrasts from lsmeans - p value adjustment

Question

I need assistance with interpreting the outcome of my pairwise comparisons from my datasets. I've been running a glmer mixed models and selecting the best model using the AIC criteria. This is the ANOVA outcome from my model which had the lowest AIC:

Anova(lm13, type="III", test.statistic = "Chisq")

Analysis of Deviance Table (Type III Wald chisquare tests)

Response: Caspase3
          Chisq Df Pr(>Chisq)    
(Intercept)  9.6287  1  0.0019156 ** 
 Post         1.8869  1  0.1695546    
 Adult        2.0416  1  0.1530531    
 Pre          7.4014  1  0.0065174 ** 
 Age          1.4426  1  0.2297269    
 Post:Adult   0.9421  1  0.3317506    
 Post:Pre     2.7315  1  0.0983886 .  
 Adult:Pre   12.0664  1  0.0005134 ***
 ---
 Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

I then use lsmeans to compute the pairwise comparisons from the contrasts.

Pre_Adult_PC <- lsmeans(lm13, ~Pre*Adult)
Pre_Adult_PC

Pre Adult     lsmean        SE df  asymp.LCL  asymp.UCL
C   C     -0.3979339 0.1174724 NA -0.6281756 -0.1676922
B   C     -0.5141986 0.1183483 NA -0.7461569 -0.2822402
C   F-    -0.5248242 0.1196266 NA -0.7592880 -0.2903604
B   F-    -0.3726527 0.1176916 NA -0.6033240 -0.1419813

Results are averaged over the levels of: Post, Age 
Results are given on the log (not the response) scale. 
Confidence level used: 0.95 

contrast(Pre_Adult_PC, alpha=0.05, method="pairwise", adjust=NULL)

contrast       estimate         SE df z.ratio p.value
C,C - B,C    0.11626466 0.05302493 NA   2.193  0.0283
C,C - C,F-   0.12689028 0.05373435 NA   2.361  0.0182
C,C - B,F-  -0.02528124 0.05236930 NA  -0.483  0.6293
B,C - C,F-   0.01062562 0.05679222 NA   0.187  0.8516
B,C - B,F-  -0.14154590 0.05483311 NA  -2.581  0.0098
C,F- - B,F- -0.15217152 0.05686818 NA  -2.676  0.0075

I read that for mixed models where the distribution is not normal, Tukey's HSD isn't advised. This would appear to be true given how highly significant the interaction between Pre and Adult is. When I put the Tukey's HSD adjustment in, I lose nearly all the significant pairwise comparisons:

 contrast(Pre_Adult_PC, alpha=0.05, method="pairwise", adjust="Tukey")

 contrast       estimate         SE df z.ratio p.value
 C,C - B,C    0.11626466 0.05302493 NA   2.193  0.1252
 C,C - C,F-   0.12689028 0.05373435 NA   2.361  0.0847
 C,C - B,F-  -0.02528124 0.05236930 NA  -0.483  0.9629
 B,C - C,F-   0.01062562 0.05679222 NA   0.187  0.9977
 B,C - B,F-  -0.14154590 0.05483311 NA  -2.581  0.0484
 C,F- - B,F- -0.15217152 0.05686818 NA  -2.676  0.0374

So for the time being I've specified the adjustment as NULL (I've looked at SIDAK adjustment too but still lose these signficant pariwise comparisons). Looking at my data visually, the data appear to match the stats when the NULL adjustment is applied.

I would like some advice as to whether or not I should report these contrasts and if so how I report them, considering there is no post-hoc adjustment. ie. do I say: "all pariwise comparisons were computed from the contrasts between factors using lsmeans package".

Any help would be greatly appreciated. I hope I'm going about this the right way!

Thanks!

Dave

Answer 1

The changes in p-values you are showing don't seem unusual. Essentially, in your case, you need a p-value of < 0.01 before adjustment to remain < 0.05 after adjustment.

In my opinion, whether or not you apply a p-value adjustment depends on how conservative you want to be with the comparisons of groups. If you want to maximize the chances of identifying potential "real" differences, you would apply no adjustment. If you wish to be more conservative, and avoid "false positive" differences, you would apply an adjustment. If you want to be really conservative, use Bonferroni. If you want to be less conservative, use a less conservative adjustment.

(I know some statisticians who would want to pull their hair out with my language here, but I'm trying to keep the point simple.)

For me, that's all there is: It's up to you how you want to balance "false positives" and "false negatives". There are situations where it makes sense to be conservative and situations where it makes sense to be more liberal.