Timeline for Do we need a global test before post hoc tests?
Current License: CC BY-SA 3.0
11 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 12, 2022 at 10:48 | comment | added | Federico Tedeschi | @Bonferroni If you have a significant global p-value, you have evidence that there is an effect whatsoever. I think this legitimates further analysis. I think the suspect of data snooping is rather natural instead, in case of non-significance of an "omnibus" test. If one person is interested in a specific association, considered as the primary goal of the analysis, then I think such association should be left out from such global test. | |
| Aug 6, 2016 at 7:28 | comment | added | Bonferroni | @whuber Actually, you said "Applying a global test first is pretty solid protection against the risk of (even inadvertently) uncovering spurious 'significant' results from post-hoc data snooping." Given that you referred to applying the global test "first" to protect against Type I errors, it sure seems like you were implying that the global test protects the individual tests that follow (a common misconception). And given that you called the global test "pretty solid protection," that seems to suggest additional protection isn't necessary. Maybe you should clarify these points in your answer. | |
| Jul 19, 2016 at 19:06 | comment | added | whuber♦ | @Bonferroni This sounds like a fair warning, but the logic of your statement is elusive. If one null is false, then the concept of a global "false positive" doesn't even apply: we're into false negative territory at that point. It sounds like you might be concerned with the individual false positive rate (which is good) but are assuming that any post-hoc tests are not corrected for interdependence and multiple comparisons. I did not mean to say--nor did I say--that you only need to perform a global test. | |
| Jul 19, 2016 at 15:16 | comment | added | Bonferroni | Whuber is incorrect that "Applying a global test first is pretty solid protection against the risk of (even inadvertently) uncovering spurious "significant" results from post-hoc data snooping." If there are many hypotheses and one null is false, the other tests aren't protected. | |
| Sep 25, 2015 at 2:49 | comment | added | Perlnika | @Speldosa interesting point, I would be curious about that too | |
| Feb 27, 2015 at 1:58 | comment | added | Speldosa | Well, couldn't you at least surmise that there was a difference between the two means with the largest difference between them, since the null hypothesis of the ANOVA is that at least one pair of means differs from each other? | |
| Feb 1, 2014 at 23:15 | comment | added | Karl Ove Hufthammer |
@amoba and @whuber: You probably know this, but I’d like to clarify it anyway. Note that it’s possible for the ANOVA test to be significant even if none of the Tukey’s HSD tests are. Simple R example with a balanced data set with three groups: set.seed(249); group = rep(1:3, each=2); y = group + rnorm(6); mod = aov(y~factor(group)); summary(mod); TukeyHSD(mod); plot(y~group)
|
|
| Jan 22, 2014 at 16:27 | comment | added | amoeba | Thank you, @whuber. I tried to find a discussion of this point here on CrossValidated, but to no avail. So I posted my own question about how is such a situation possible: stats.stackexchange.com/questions/83030/…. I would be very grateful indeed if you could elaborate there! | |
| Jan 22, 2014 at 16:21 | comment | added | whuber♦ | @amoeba That's right; I am referring to unadjusted pairwise tests. Thank you for clarifying this point. | |
| Jan 22, 2014 at 10:35 | comment | added | amoeba | Re (2): when you say that a one-way ANOVA can report a significant difference when none of the pairwise t-tests does, do you refer to simple non-adjusted ("non post", e.g. not Tukey's procedure or anything) t-tests? I thought this would never be possible, was I wrong? | |
| Apr 19, 2011 at 17:22 | history | answered | whuber♦ | CC BY-SA 3.0 |