Questions tagged [post-hoc]

"post-hoc" refers to analyses that are decided upon after the data has been collected, as opposed to "a priori".

Filter by
Sorted by
Tagged with
63
votes
2answers
24k views

Do we need a global test before post hoc tests?

I often hear that post hoc tests after an ANOVA can only be used if the ANOVA itself was significant. However, post hoc tests adjust $p$-values to keep the global type I error rate at 5%, don't they?...
40
votes
2answers
10k views

What are the practical differences between the Benjamini & Hochberg (1995) and the Benjamini & Yekutieli (2001) false discovery rate procedures?

My statistics program implements both the Benjamini & Hochberg (1995) and Benjamini & Yekutieli (2001) false discovery rate (FDR) procedures. I have done my best to read through the later ...
32
votes
3answers
13k views

Can ANOVA be significant when none of the pairwise t-tests is?

Is it possible for one-way (with $N>2$ groups, or "levels") ANOVA to report a significant difference when none of the $N(N-1)/2$ pairwise t-tests does? In this answer @whuber wrote: It is ...
26
votes
3answers
58k views

Post hoc test after ANOVA with repeated measures using R

I have performed a repeated measures ANOVA in R, as follows: aov_velocity = aov(Velocity ~ Material + Error(Subject/(Material)), data=scrd) summary(aov_velocity) ...
23
votes
4answers
37k views

Is Fisher's LSD as bad as they say it is?

When we perform experiments (on small sample sizes (usually the sample size per treatment group is about 7~8)) on two groups, we use a t-test to test for difference. However, when we perform an ANOVA (...
22
votes
1answer
96k views

Bonferroni or Tukey? When does the number of comparisons become large?

Reading Field's Discovering Statistics Using SPSS (3rd Edition) I was struck by a bit about post-hoc tests in ANOVA. For those wanting to control the Type I error rate he suggests Bonferroni or Tukey ...
21
votes
1answer
62k views

Post-hoc tests after Kruskal-Wallis: Dunn's test or Bonferroni corrected Mann-Whitney tests?

I have some non-Gaussian distributed variable and I need to check if there are significant differences between the values of this variable in 5 different groups. I have performed Kruskal-Wallis one-...
21
votes
3answers
53k views

How can I get a significant overall ANOVA but no significant pairwise differences with Tukey's procedure?

I performed with R an ANOVA and I got significant differences. However when checking which pairs were significantly different using the Tukey's procedure I did not get any of them. How can this be ...
20
votes
5answers
22k views

Post-hocs for within subjects tests?

What is the preferred method for for conducting post-hocs for within subjects tests? I've seen published work where Tukey's HSD is employed but a review of Keppel and Maxwell & Delaney suggests ...
20
votes
3answers
44k views

How to perform post-hoc test on lmer model?

This is my data frame: ...
20
votes
1answer
6k views

Why use ANOVA at all instead of jumping straight into post-hoc or planned comparisons tests?

Looking at a between-groups ANOVA situation, what do you get by actually doing such an ANOVA test first, and second do post-hoc (Bonferroni, Šidák, etc.) or planned comparisons tests? Why not skip the ...
18
votes
2answers
156k views

How do you write up Tukey post-hoc findings?

What is the proper way to write a Tukey post-hoc result? There are several examples with different results? Say you have North, South, East and West. ...
17
votes
2answers
12k views

Post-hoc test for chi-square goodness-of-fit test

I'm conducting a chi-square goodness-of-fit (GOF) test with three categories and specifically want to test the null that the population proportions in each category are equal (i.e., the proportion is ...
16
votes
2answers
18k views

Which multiple comparison method to use for a lmer model: lsmeans or glht?

I'm analyzing a data set using a mixed effects model with one fixed effect (condition) and two random effects (participant due to the within subject design and pair). The model was generated with the <...
15
votes
4answers
51k views

Sidak or Bonferroni?

I am using a generalized linear model in SPSS to look at the differences in average number of caterpillars (non-normal, using Tweedie distribution) on 16 different species of plants. I want to run ...
15
votes
2answers
104k views

How to obtain the results of a Tukey HSD post-hoc test in a table showing grouped pairs?

I would love to perform a TukeyHSD post-hoc test after my two-way Anova with R, obtaining a table containing the sorted pairs grouped by significant difference. (Sorry about the wording, I'm still new ...
15
votes
6answers
5k views

What is the problem with post-hoc testing?

My statistic professor says so, all the books that I look at state it: post-hoc-testing is unscientific. You must derive a hypothesis from theory first, and then collect data and analyse it. But I ...
14
votes
3answers
12k views

How to correctly apply the Nemenyi post-hoc test after the Friedman test

I'm comparing the performance of multiple algorithms on multiple data sets. Since those performance measurements are not guaranteed to be normally distributed, I chose the Friedman Test with the ...
12
votes
3answers
2k views

Is publishing work based on post hoc analysis problematic? [duplicate]

Nowadays, in the scientific domain, increasingly, people in the top position argue against doing post-hoc analysis and they advise against collecting a bunch of data and just make up a story after ...
12
votes
2answers
30k views

How to perform post-hoc comparison on interaction term with mixed-effects model?

I’m working on a data set in order to evaluate the impact of drying on sediment microbial activities. The objective is to determine if the impact of drying varies among sediment types and/or depth ...
11
votes
3answers
11k views

Friedman test and post-hoc test for Python

In my dataset, I have five (ordinal) groups with an x-amount of measurement. Because homoscedasticity is violated, I performed the Friedman chi-square test to see if there are any statistical ...
11
votes
1answer
17k views

Post-hoc testing in multcomp::glht for mixed-effects models (lme4) with interactions

I am performing post-hoc tests on a linear mixed-effects model in R (lme4 package). I am using ...
11
votes
1answer
415 views

If we shouldn't do post hoc power calculations, are post hoc effect size calculations also invalid?

We have posts on here discussing why post hoc power calculations are flawed. What about post hoc calculations of the effect size that we could catch? That seems logically equivalent. If we are not ...
11
votes
2answers
10k views

What is a valid post-hoc analysis for a three-way repeated measures ANOVA?

I've performed a three-way repeated measures ANOVA; what post-hoc analyses are valid? This is a fully balanced design (2x2x2) with one of the factors having a within-subjects repeated measure. I'm ...
9
votes
4answers
19k views

Can Mann-Whitney test be used for post-hoc comparisons after Kruskal-Wallis?

I have a simulation where an animal is placed in a hostile environment and timed to see how long it can survive using some approach to survival. There are three approaches it can use to survive. I ran ...
9
votes
2answers
29k views

Post-hoc tests for MANOVA: univariate ANOVAs or discriminant analysis?

I am using a MANOVA test to compare nine different dependent variables (from neuropsychological and neuropsychiatric assessment) between three groups. The output shows a significant influence from ...
9
votes
1answer
3k views

How could a Tukey HSD test be more signif then the uncorrected P value of t.test?

I came by the post "Post-hoc Pairwise Comparisons of Two-way ANOVA" (responding to this post), which shows the following: ...
9
votes
1answer
502 views

It's all in the family; but do we include the in-laws too?

Suppose I have an experiment with two or more factors. An overall ANOVA is constructed, and then we follow-up with two or more sets of post hoc tests, say multiple comparisons. My question is about ...
9
votes
2answers
10k views

Post-hoc test after 2-factor repeated measures ANOVA in R?

I have problems finding a solution regarding how to run a post-hoc test (Tukey HSD) after a 2-factor (both within-subjects) repeated-measures ANOVA in R. For the ANOVA, I've used the aov -function: <...
8
votes
1answer
4k views

Multiple comparisons with many groups

I would like to determine if using multiple comparisons test would be appropriate for my data. I used the Kruskal-Wallis test to determine if there were differences in mean inhibition among $17$ ...
8
votes
3answers
4k views

Simple post-stratification weights in R

I just got my hands on the ANES (American National Election Studies) 2008 data set, and would like to do some simple analysis in R. However, I've never worked with this complex of a data set before ...
8
votes
1answer
9k views

Which post-hoc is more valid for multiple comparison of an unbalanced lmer-model: lsm or mcp?

After doing a model comparison with my mixed lmer model, I have a model with three main effects, no interaction, say ...
8
votes
3answers
4k views

Is there a consensus on adjusting alpha for multiple contrasts if the main effect is significant?

After talking to a couple of statisticians, reading some book sections, internet articles, and forums like this, I am still unclear about multiplicity adjustments of post hoc tests and contrasts. Let'...
8
votes
3answers
26k views

Post hoc test in a 2x3 mixed design ANOVA using SPSS?

I have two groups of 10 participants who were assessed three times during an experiment. To test for differences between groups and across the three assessments, I ran a 2x3 mixed design ANOVA with <...
8
votes
2answers
246 views

Are there realistic/relevant use-cases for one way ANOVA?

I have trouble imagining realistic research questions that can be answered by a one way ANOVA. Two or more ways with interaction terms and perhaps mixed factors (within and between subjects), sure you ...
8
votes
2answers
6k views

Post hoc test of adjusted means (after ANCOVA)

When doing ANCOVAs, software usually gives both unadjusted means as well as adjusted means. However, the post hoc tests for most statistical programs are limited to unadjusted means. Yet, sometimes, ...
7
votes
2answers
11k views

How to interpret a post hoc Tukey's test?

I did a one-way ANOVA followed by a Tukey's test to compare the means of different treatments. Let's say the treatments are A, B and C. The table of multiple comparisons tells me there is a ...
7
votes
2answers
24k views

lsmeans (R): Adjust for multiple comparisons with interaction terms

I have a lsmeans problem in R. I want to do a post-hoc analysis of an interaction, similar to examples provided in the lsmeans documentation. I am puzzled by the fact that the p-values are the same ...
7
votes
4answers
2k views

Strange result of post-hoc test

I have data for a test on three groups. The measured variable is ratio scaled. The R code is g1a<-c(7, 3, 40) g2a<-c(1,1,2) g3a<-c(0,0,0) Since the ...
7
votes
4answers
15k views

Friedman's test is very significant, but its post hoc comparisons (SPSS) are not significant

I ran a non-parametric Friedman's test for my data in SPSS 22 and significantly rejected the null. That would mean that among the $k$ paired samples (3 in my case), there should be detected at least ...
7
votes
3answers
17k views

ANOVA results do not match post-hoc Tukey test, how to proceed?

There has been some discussion about why differences exist between factor significance after an ANOVA and no significance in the pairwise comparisons, and it has been very helpful. However, there is ...
7
votes
1answer
2k views

Correcting for multiple pairwise comparisons with GAM objects {mgcv} in R

My question concerns pairwise comparisons of factor levels in a gam object. I have a dataframe, df, containing reaction time data (in ms) to stimuli varying in ...
7
votes
3answers
52k views

Minimum sample size for 1 way ANOVA?

I have tested a group of participants over various time points and I want to investigate whether mean score on each subsequent time point is significantly different to the mean score at baseline (eg. ...
7
votes
2answers
3k views

Analyzing repeated measures experiment with multiple treatment groups and multiple measures

I am doing a study to evaluate the effect of two dietary supplements on body composition. Literature suggests that Treatment 1 (Dietary Supplement 1) affects body composition with some side effects on ...
7
votes
3answers
6k views

Post hoc $\chi^2$ test with R

This is a quite simple question but I don't find any good, clear, precise answers: I'm looking for a way to perform post hoc test on a chi$^2$ test. I have 2 variables : var1 : good/fair/poor and ...
7
votes
1answer
348 views

Post-hoc test to determine difference in variance

I'm testing whether 5 different groups have different variances, I am not interested in the mean. I performed a Brown-Forsythe test given that the data does not fit a normal distribution. The test ...
7
votes
1answer
2k views

Post-hoc test in two-way ANOVA

I am doing a two-way independent ANOVA using type of organisational change (5 levels) and job level (7 levels) within the organisation as the factors (in SPSS). I followed Andy Field's procedure (...
6
votes
2answers
8k views

What is the difference between various Kruskal-Wallis post-hoc tests?

[There seem to be a lot of similar questions here, so please point me the right direction if this has already been answered, but I think it's reasonably differentiated.] There are many different ...
6
votes
3answers
2k views

Meaning of “post hoc” multiple comparisons

My original thought was “post hoc” in a post hoc multiple comparison procedure means that the procedure is taken after an omnibus test (such as ANOVA F test) gives a significant result. But after ...
6
votes
2answers
10k views

How to get only desirable comparisons from post-hoc

I have some experimental data that I'm trying to analyze. I have 1 response variable and 3 explanatory variables (these are factor variables). The explanatory variables are the presence of a disease(...

1
2 3 4 5
12