30
votes
Accepted
How to perform post-hoc test on lmer model?
You could use emmeans::emmeans() or lmerTest::difflsmeans(), or multcomp::glht().
I prefer <...
- 801
26
votes
Accepted
Which multiple comparison method to use for a lmer model: lsmeans or glht?
Not a complete answer...
The difference between glht(myfit, mcp(myfactor="Tukey")) and the two other methods is that this way uses a "z" statistic (normal ...
- 18.6k
15
votes
Accepted
Is publishing work based on post hoc analysis problematic?
Obviously post-hoc analysis and explanatory studies provide useful pieces to understand and discuss phenomena : There is nothing wrong with them and with publishing them. The moment where such ...
- 4,903
14
votes
How to perform post-hoc test on lmer model?
After you've fit your lmer model you can do ANOVA, MANOVA, and multiple comparison procedures on the model object, like this:
...
- 1,015
13
votes
Accepted
Correcting for multiple pairwise comparisons with GAM objects {mgcv} in R
The glht() function for generalized linear hypotheses from the multcomp package can be used to carry out various kinds of ...
- 15.2k
13
votes
Is it ok to run post hoc comparisons if ANOVA is nearly significant?
Since multiple comparison tests are often called 'post tests', you'd think they logically follow the one-way ANOVA and should be used only when the overall ANOVA results in $p < 0.05$ (or whatever ...
- 19.7k
11
votes
Accepted
Friedman's test is very significant, but its post hoc comparisons (SPSS) are not significant
SPSS Algorithms state that in doing pairwise comparisons after Friedman test they use the Dunn's (1964) procedure. I didn't read that Dunn's original paper so I can't say if SPSS follows it correctly, ...
- 56.5k
11
votes
Accepted
Zero inflated beta regression using gamlss for vegetation cover data
I have added preliminary support for gamlss to the emmeans package...
...
- 19.4k
11
votes
Non-significant results when running Kruskal-Wallis, significant results when running Dwass-Steel-Critchlow-Flinger pairwise comparisons
My question is - is DSCF a post-hoc test which is supposed to only be applied when statistically significant differences are found on Kruskal-Wallis?
Yes, the Kruskal-Wallis test is a non-parametric ...
- 56.8k
10
votes
Accepted
Friedman test and post-hoc test for Python
I am currently looking into this issue myself; according to this paper there are a number of possibilities to perform posthoc-tests (Update: an extension regarding the use of non-parametric tests can ...
- 595
10
votes
Accepted
Repeated measures ANOVA with significant interaction effect, but non-significant main effect
Inadvertently, you're comparing apples and oranges. Once an interaction is included in the model, the interpretation of the main effect changes. With the interaction, the main effect of "...
- 60.6k
10
votes
Accepted
Is it ok to run post hoc comparisons if ANOVA is nearly significant?
I highly recommend reading Midway et al., 2020, which is probably the best article I have ever read that summarizes pairwise comparisons for ANOVA. Along with the guidelines they provide for how to ...
- 8,645
9
votes
Is Fisher's LSD as bad as they say it is?
The reasoning behind Fisher's LSD can be extended to cases beyond N=3.
I'll discuss the case of four groups in detail. To keep the familywise Type-I error rate at 0.05 or below, a multiple-comparison ...
- 91
9
votes
Minimum sample size for 1 way ANOVA?
Described below are three approaches to estimating sample size for completely randomized designs. Note that the procedures differ in terms of the information you must provide.
Approach #1 (requires ...
- 129
8
votes
Accepted
Easy post-hoc tests when meta-analyzing with the `metafor` package in r
You could use the contrMat() function. Something like this should work:
...
- 16.7k
8
votes
Accepted
Why do planned comparisons and post-hoc tests differ?
They aren't really the same. A planned comparison is something you are committing to before you see your data, and will run no matter what the results look like. A post-hoc comparison is more ...
8
votes
Accepted
Report p-values from model fitting or glht?
There are two things going on here:
the difference between t-tests and Z-tests (as pointed out by @vkehayas); t-tests account for the uncertainty in the estimate of the standard error, so should be ...
- 41.6k
8
votes
Accepted
Interpreting the standard error from emmeans - R
OK, let us dissect this model. First, the model itself:
...
- 19.4k
8
votes
Accepted
If we shouldn't do post hoc power calculations, are post hoc effect size calculations also invalid?
The problem is in the use of the "post-hoc effect size," not that its calculation is invalid. A "post-hoc effect size" is fundamentally an estimate of population parameters (e.g., ...
- 86.8k
8
votes
Accepted
Discrepancy between the results of the Anova and the post-hoc test
The OP posed a very similar question later on, which I link here for completeness as it has interesting & complementary answer(s): Discrepancy between the results of the ANOVA and the post-hoc ...
- 8,790
7
votes
What are the practical differences between the Benjamini & Hochberg (1995) and the Benjamini & Yekutieli (2001) false discovery rate procedures?
p.adjust is not misciting for BY. The reference is to Theorem 1.3 (proof in Section 5 on p.1182) in the paper:
Benjamini, Y., and Yekutieli, D. (2001). The control of the false discovery rate in ...
- 461
7
votes
Post hoc test after ANOVA with repeated measures using R
If you want to stick with the aov() function you can use the emmeans package which can handle ...
- 5,165
7
votes
Accepted
Analyzing repeated measures experiment with multiple treatment groups and multiple measures
I think one could write a whole book dealing exclusively with your question (and I am definitely not qualified to write it). So without any attempt at providing a comprehensive answer, here are some ...
- 103k
7
votes
Accepted
Post hoc power analysis for a non significant result?
Power analyses exploit an equation with four variables ($\alpha$, power, $N$, and the effect size). When you solve for power by stipulating the others, it is called "post hoc" power analysis. People ...
7
votes
Is it possible to find non-significant result from one-way ANOVA but significant results from individual post-hoc tests?
Yes, it is possible for the omnibus ANOVA test statistic (testing the null hypothesis that the data arise from groups with the same mean) to be non-significant, while individual tests (allowing for ...
- 56.8k
7
votes
Post Hoc Pairwise Comparison of Interaction in Mixed Effects (lmer) Model
Try the emmeans package. Something like
...
- 19.4k
7
votes
In R, how can the p-value during pairwise t tests without adjustment be different than the p-value of the t test alone?
Because the first one fits a model to the data for all the treatments and thus uses all the data to estimate the error variance; whereas the manually done test uses the data from only those two ...
- 19.4k
7
votes
Accepted
What is more powerful – an ANOVA test or post hoc tests?
To make anova and posthoc tests comparable, you need to be conducting posthoc tests for at least $G-1$ contrasts (where $G$ is the number of groups) such that the contrasts span the space of all ...
- 12.1k
7
votes
Accepted
Post-hoc power analysis for null results: how to use 95% confidence interval instead?
If your CIs are narrow, then you have an idea of how large the effect is, and you can say with some confidence that the effect is small, and that's why you didn't detect it.
If the CIs are wide, then ...
- 16.5k
7
votes
Non-significant results when running Kruskal-Wallis, significant results when running Dwass-Steel-Critchlow-Flinger pairwise comparisons
What is ANOVA?
It is important to note some of the reasons that ANOVA exist, as it can provide some context about how you should consider your omnibus ANOVA and the pairwise tests that accompany them (...
- 8,645
Only top scored, non community-wiki answers of a minimum length are eligible