Pairwise comparisons of contrasts from lsmeans - p value adjustment

I need assistance with interpreting the outcome of my pairwise comparisons from my datasets. I've been running a glmer mixed models and selecting the best model using the AIC criteria. This is the ANOVA outcome from my model which had the lowest AIC:

Anova(lm13, type="III", test.statistic = "Chisq")

Analysis of Deviance Table (Type III Wald chisquare tests)

Response: Caspase3
Chisq Df Pr(>Chisq)
(Intercept)  9.6287  1  0.0019156 **
Post         1.8869  1  0.1695546
Pre          7.4014  1  0.0065174 **
Age          1.4426  1  0.2297269
Post:Pre     2.7315  1  0.0983886 .
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1


I then use lsmeans to compute the pairwise comparisons from the contrasts.

Pre_Adult_PC <- lsmeans(lm13, ~Pre*Adult)

Pre Adult     lsmean        SE df  asymp.LCL  asymp.UCL
C   C     -0.3979339 0.1174724 NA -0.6281756 -0.1676922
B   C     -0.5141986 0.1183483 NA -0.7461569 -0.2822402
C   F-    -0.5248242 0.1196266 NA -0.7592880 -0.2903604
B   F-    -0.3726527 0.1176916 NA -0.6033240 -0.1419813

Results are averaged over the levels of: Post, Age
Results are given on the log (not the response) scale.
Confidence level used: 0.95

contrast       estimate         SE df z.ratio p.value
C,C - B,C    0.11626466 0.05302493 NA   2.193  0.0283
C,C - C,F-   0.12689028 0.05373435 NA   2.361  0.0182
C,C - B,F-  -0.02528124 0.05236930 NA  -0.483  0.6293
B,C - C,F-   0.01062562 0.05679222 NA   0.187  0.8516
B,C - B,F-  -0.14154590 0.05483311 NA  -2.581  0.0098
C,F- - B,F- -0.15217152 0.05686818 NA  -2.676  0.0075


I read that for mixed models where the distribution is not normal, Tukey's HSD isn't advised. This would appear to be true given how highly significant the interaction between Pre and Adult is. When I put the Tukey's HSD adjustment in, I lose nearly all the significant pairwise comparisons:

 contrast(Pre_Adult_PC, alpha=0.05, method="pairwise", adjust="Tukey")

contrast       estimate         SE df z.ratio p.value
C,C - B,C    0.11626466 0.05302493 NA   2.193  0.1252
C,C - C,F-   0.12689028 0.05373435 NA   2.361  0.0847
C,C - B,F-  -0.02528124 0.05236930 NA  -0.483  0.9629
B,C - C,F-   0.01062562 0.05679222 NA   0.187  0.9977
B,C - B,F-  -0.14154590 0.05483311 NA  -2.581  0.0484
C,F- - B,F- -0.15217152 0.05686818 NA  -2.676  0.0374


So for the time being I've specified the adjustment as NULL (I've looked at SIDAK adjustment too but still lose these signficant pariwise comparisons). Looking at my data visually, the data appear to match the stats when the NULL adjustment is applied.

I would like some advice as to whether or not I should report these contrasts and if so how I report them, considering there is no post-hoc adjustment. ie. do I say: "all pariwise comparisons were computed from the contrasts between factors using lsmeans package".

Thanks!

Dave

• Failing to do any multiplicity adjustment at all is a sorry response to some nebulous concern about non-normality. The Tukey adjustment will not be exact, but it'll be reasonable. The goal of statistics isn't to maximize the number of asterisks. It's to make honest conclusions about what is discernible in the data. – Russ Lenth Jul 29 '17 at 2:58
• Yes I totally understand this and it isn't my intention to produce misleading results. I just to wish to know what the best approach is and if I'm choosing a post-hoc adjustment, which is the best one to choose based on my model. Thanks! – Dave Walkr Jul 29 '17 at 18:25
• A Bonferroni adjustment is always appropriate - though somewhat conservative. – Russ Lenth Jul 30 '17 at 13:40

The changes in p-values you are showing don't seem unusual. Essentially, in your case, you need a p-value of < 0.01 before adjustment to remain < 0.05 after adjustment.

In my opinion, whether or not you apply a p-value adjustment depends on how conservative you want to be with the comparisons of groups. If you want to maximize the chances of identifying potential "real" differences, you would apply no adjustment. If you wish to be more conservative, and avoid "false positive" differences, you would apply an adjustment. If you want to be really conservative, use Bonferroni. If you want to be less conservative, use a less conservative adjustment.

(I know some statisticians who would want to pull their hair out with my language here, but I'm trying to keep the point simple.)

For me, that's all there is: It's up to you how you want to balance "false positives" and "false negatives". There are situations where it makes sense to be conservative and situations where it makes sense to be more liberal.

• Hi, thanks for your help Sal. I think probably employing an adjustment is the safe way to go but I'm not sure which one I should choose based on my model. – Dave Walkr Jul 29 '17 at 18:26
• My best advice is read the section on adjustments in the documentation for the summary function in lsmeans. It doesn't give much guidance on when each adjustment method would be preferred, but may be helpful. – Sal Mangiafico Jul 30 '17 at 10:15