# Pairwise comparisons of contrasts from lsmeans - p value adjustment

I need assistance with interpreting the outcome of my pairwise comparisons from my datasets. I've been running a glmer mixed models and selecting the best model using the AIC criteria. This is the ANOVA outcome from my model which had the lowest AIC:

Anova(lm13, type="III", test.statistic = "Chisq")

Analysis of Deviance Table (Type III Wald chisquare tests)

Response: Caspase3
Chisq Df Pr(>Chisq)
(Intercept)  9.6287  1  0.0019156 **
Post         1.8869  1  0.1695546
Pre          7.4014  1  0.0065174 **
Age          1.4426  1  0.2297269
Post:Pre     2.7315  1  0.0983886 .
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1


I then use lsmeans to compute the pairwise comparisons from the contrasts.

Pre_Adult_PC <- lsmeans(lm13, ~Pre*Adult)

Pre Adult     lsmean        SE df  asymp.LCL  asymp.UCL
C   C     -0.3979339 0.1174724 NA -0.6281756 -0.1676922
B   C     -0.5141986 0.1183483 NA -0.7461569 -0.2822402
C   F-    -0.5248242 0.1196266 NA -0.7592880 -0.2903604
B   F-    -0.3726527 0.1176916 NA -0.6033240 -0.1419813

Results are averaged over the levels of: Post, Age
Results are given on the log (not the response) scale.
Confidence level used: 0.95

contrast       estimate         SE df z.ratio p.value
C,C - B,C    0.11626466 0.05302493 NA   2.193  0.0283
C,C - C,F-   0.12689028 0.05373435 NA   2.361  0.0182
C,C - B,F-  -0.02528124 0.05236930 NA  -0.483  0.6293
B,C - C,F-   0.01062562 0.05679222 NA   0.187  0.8516
B,C - B,F-  -0.14154590 0.05483311 NA  -2.581  0.0098
C,F- - B,F- -0.15217152 0.05686818 NA  -2.676  0.0075


I read that for mixed models where the distribution is not normal, Tukey's HSD isn't advised. This would appear to be true given how highly significant the interaction between Pre and Adult is. When I put the Tukey's HSD adjustment in, I lose nearly all the significant pairwise comparisons:

 contrast(Pre_Adult_PC, alpha=0.05, method="pairwise", adjust="Tukey")

contrast       estimate         SE df z.ratio p.value
C,C - B,C    0.11626466 0.05302493 NA   2.193  0.1252
C,C - C,F-   0.12689028 0.05373435 NA   2.361  0.0847
C,C - B,F-  -0.02528124 0.05236930 NA  -0.483  0.9629
B,C - C,F-   0.01062562 0.05679222 NA   0.187  0.9977
B,C - B,F-  -0.14154590 0.05483311 NA  -2.581  0.0484
C,F- - B,F- -0.15217152 0.05686818 NA  -2.676  0.0374


So for the time being I've specified the adjustment as NULL (I've looked at SIDAK adjustment too but still lose these signficant pariwise comparisons). Looking at my data visually, the data appear to match the stats when the NULL adjustment is applied.

I would like some advice as to whether or not I should report these contrasts and if so how I report them, considering there is no post-hoc adjustment. ie. do I say: "all pariwise comparisons were computed from the contrasts between factors using lsmeans package".

Thanks!

Dave

• Failing to do any multiplicity adjustment at all is a sorry response to some nebulous concern about non-normality. The Tukey adjustment will not be exact, but it'll be reasonable. The goal of statistics isn't to maximize the number of asterisks. It's to make honest conclusions about what is discernible in the data. – rvl Jul 29 '17 at 2:58
• Yes I totally understand this and it isn't my intention to produce misleading results. I just to wish to know what the best approach is and if I'm choosing a post-hoc adjustment, which is the best one to choose based on my model. Thanks! – Dave Walkr Jul 29 '17 at 18:25
• A Bonferroni adjustment is always appropriate - though somewhat conservative. – rvl Jul 30 '17 at 13:40