Skip to main content
31 votes
Accepted

How to perform post-hoc test on lmer model?

You could use emmeans::emmeans() or lmerTest::difflsmeans(), or multcomp::glht(). I prefer <...
Kayle Sawyer's user avatar
16 votes
Accepted

Is publishing work based on post hoc analysis problematic?

Obviously post-hoc analysis and explanatory studies provide useful pieces to understand and discuss phenomena : There is nothing wrong with them and with publishing them. The moment where such ...
beuhbbb's user avatar
  • 5,063
14 votes

How to perform post-hoc test on lmer model?

After you've fit your lmer model you can do ANOVA, MANOVA, and multiple comparison procedures on the model object, like this: ...
Hack-R's user avatar
  • 1,015
13 votes
Accepted

Correcting for multiple pairwise comparisons with GAM objects {mgcv} in R

The glht() function for generalized linear hypotheses from the multcomp package can be used to carry out various kinds of ...
Achim Zeileis's user avatar
13 votes

Is it ok to run post hoc comparisons if ANOVA is nearly significant?

Since multiple comparison tests are often called 'post tests', you'd think they logically follow the one-way ANOVA and should be used only when the overall ANOVA results in $p < 0.05$ (or whatever ...
Harvey Motulsky's user avatar
12 votes

Non-significant results when running Kruskal-Wallis, significant results when running Dwass-Steel-Critchlow-Flinger pairwise comparisons

My question is - is DSCF a post-hoc test which is supposed to only be applied when statistically significant differences are found on Kruskal-Wallis? Yes, the Kruskal-Wallis test is a non-parametric ...
Robert Long's user avatar
  • 63.9k
11 votes
Accepted

Zero inflated beta regression using gamlss for vegetation cover data

I have added preliminary support for gamlss to the emmeans package... ...
Russ Lenth's user avatar
  • 20.7k
10 votes
Accepted

Friedman test and post-hoc test for Python

I am currently looking into this issue myself; according to this paper there are a number of possibilities to perform posthoc-tests (Update: an extension regarding the use of non-parametric tests can ...
Archie's user avatar
  • 595
10 votes
Accepted

Repeated measures ANOVA with significant interaction effect, but non-significant main effect

Inadvertently, you're comparing apples and oranges. Once an interaction is included in the model, the interpretation of the main effect changes. With the interaction, the main effect of "...
AdamO's user avatar
  • 63.7k
10 votes
Accepted

Is it ok to run post hoc comparisons if ANOVA is nearly significant?

I highly recommend reading Midway et al., 2020, which is probably the best article I have ever read that summarizes pairwise comparisons for ANOVA. Along with the guidelines they provide for how to ...
Shawn Hemelstrand's user avatar
9 votes

Friedman's test is very significant, but its post hoc comparisons (SPSS) are not significant

I found (via a ResearchGate question) lots of good stuff in the vignette for the PMCMR package (which is now deprecated in favor of PMCMRplus), including post hoc ...
Nick Stauner's user avatar
  • 12.4k
9 votes
Accepted

If we shouldn't do post hoc power calculations, are post hoc effect size calculations also invalid?

The problem is in the use of the "post-hoc effect size," not that its calculation is invalid. A "post-hoc effect size" is fundamentally an estimate of population parameters (e.g., ...
EdM's user avatar
  • 95.4k
9 votes
Accepted

Post-hoc power analysis for null results: how to use 95% confidence interval instead?

If your CIs are narrow, then you have an idea of how large the effect is, and you can say with some confidence that the effect is small, and that's why you didn't detect it. If the CIs are wide, then ...
Jeremy Miles's user avatar
  • 18.3k
9 votes

Non-significant results when running Kruskal-Wallis, significant results when running Dwass-Steel-Critchlow-Flinger pairwise comparisons

What is ANOVA? It is important to note some of the reasons that ANOVA exist, as it can provide some context about how you should consider your omnibus ANOVA and the pairwise tests that accompany them (...
Shawn Hemelstrand's user avatar
8 votes
Accepted

Easy post-hoc tests when meta-analyzing with the `metafor` package in r

You could use the contrMat() function. Something like this should work: ...
Wolfgang's user avatar
  • 17.4k
8 votes
Accepted

Why do planned comparisons and post-hoc tests differ?

They aren't really the same. A planned comparison is something you are committing to before you see your data, and will run no matter what the results look like. A post-hoc comparison is more ...
gung - Reinstate Monica's user avatar
8 votes
Accepted

Report p-values from model fitting or glht?

There are two things going on here: the difference between t-tests and Z-tests (as pointed out by @vkehayas); t-tests account for the uncertainty in the estimate of the standard error, so should be ...
Ben Bolker's user avatar
  • 44.5k
8 votes
Accepted

Interpreting the standard error from emmeans - R

OK, let us dissect this model. First, the model itself: ...
Russ Lenth's user avatar
  • 20.7k
8 votes

How are the degrees of freedom in the emmeans package calculated? - R

Your t.test() results are each based on only selected portions of the data, whereas the emmeans() results are based on a model ...
Russ Lenth's user avatar
  • 20.7k
8 votes
Accepted

Discrepancy between the results of the Anova and the post-hoc test

The OP posed a very similar question later on, which I link here for completeness as it has interesting & complementary answer(s): Discrepancy between the results of the ANOVA and the post-hoc ...
dipetkov's user avatar
  • 10.5k
7 votes

What are the practical differences between the Benjamini & Hochberg (1995) and the Benjamini & Yekutieli (2001) false discovery rate procedures?

p.adjust is not misciting for BY. The reference is to Theorem 1.3 (proof in Section 5 on p.1182) in the paper: Benjamini, Y., and Yekutieli, D. (2001). The control of the false discovery rate in ...
John Maindonald's user avatar
7 votes

Post hoc test after ANOVA with repeated measures using R

If you want to stick with the aov() function you can use the emmeans package which can handle ...
statmerkur's user avatar
  • 6,325
7 votes

Is it possible to find non-significant result from one-way ANOVA but significant results from individual post-hoc tests?

Yes, it is possible for the omnibus ANOVA test statistic (testing the null hypothesis that the data arise from groups with the same mean) to be non-significant, while individual tests (allowing for ...
Robert Long's user avatar
  • 63.9k
7 votes

Is publishing work based on post hoc analysis problematic?

There's a formula called "Bayes' Theorem" that says that if you start out assigning probability P1 to a hypothesis H, and you see evidence E, then you should adjust your probability to: P2 = P1*(...
Acccumulation's user avatar
7 votes

Post Hoc Pairwise Comparison of Interaction in Mixed Effects (lmer) Model

Try the emmeans package. Something like ...
Russ Lenth's user avatar
  • 20.7k
7 votes

In R, how can the p-value during pairwise t tests without adjustment be different than the p-value of the t test alone?

Because the first one fits a model to the data for all the treatments and thus uses all the data to estimate the error variance; whereas the manually done test uses the data from only those two ...
Russ Lenth's user avatar
  • 20.7k
7 votes
Accepted

Non-significant Kruskal-Wallis and significant post hoc tests

In general, if you use an omnibus test, such as an ANOVA F-test or a Kruskal-Wallis H-test, it is illogical and poor practice to conduct pairwise comparisons when you fail to reject the null ...
LSC's user avatar
  • 813
7 votes
Accepted

What is more powerful – an ANOVA test or post hoc tests?

To make anova and posthoc tests comparable, you need to be conducting posthoc tests for at least $G-1$ contrasts (where $G$ is the number of groups) such that the contrasts span the space of all ...
Gordon Smyth's user avatar
  • 13.2k
6 votes

How can I get a significant overall ANOVA but no significant pairwise differences with Tukey's procedure?

Below is a copy from an answer to a duplicate question R Tukey HSD Anova: Anova significant, Tukey not? Since this answer is not visible here, and there is no clear link, I create this copy. The ...
Sextus Empiricus's user avatar
6 votes

Sidak or Bonferroni?

Sidak and Bonferroni are so similar that you will probably get the same result regardless of which procedure you use. Bonferroni is only marginally more conservative than Sidak. For instance, for 2 ...
Bonferroni's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible