Timeline for What is more powerful – an ANOVA test or post hoc tests?
Current License: CC BY-SA 4.0
19 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| May 3, 2021 at 12:00 | history | tweeted | twitter.com/StackStats/status/1389188027357007872 | ||
| Apr 7, 2021 at 16:17 | comment | added | Alexis | @GordonSmyth Thank you for entertaining my question about Holm's method. | |
| Apr 7, 2021 at 5:05 | comment | added | Gordon Smyth | @Alexis There are other more powerful and more specialized multiple testing procedures for the oneway layout (discussed in Peter Westfall(aka BigBendRegion)'s book), but I answered the question in terms of Holm's to keep it simple. Holm's method is the simplest and most flexible that offers strong familywise error rate control. | |
| Apr 7, 2021 at 5:01 | comment | added | Gordon Smyth | @Alexis The fact that the minimum of a number of adjusted p-values is a valid p-value for testing the intersection of null hypotheses is pretty much the definition of (weak) familywise error rate control. Yes, I could have suggested a multiple testing procedure that only offers weak familywise error rate control such as Simes method. That would be more powerful than Holm's for the overall null hypothesis but would not then offer strong error rate control for the individual tests. | |
| Apr 7, 2021 at 4:51 | comment | added | Gordon Smyth | @Alexis I am not changing or contorting the mean of statistical power in any way. There is only one definition of statistical power. As for not doing the F-test: we are comparing two procedures, one is to conduct the F-test and not t-tests, the second is to conduct a series of anova t-tests but not the F-test. | |
| Apr 6, 2021 at 23:16 | comment | added | Alexis | @GordonSmyth You have lost me on where you are contrasting "power of ANOVA" by "not conducting the F-test," and I think you are contorting the meaning of "power of a test." (Aside: why would you use Holm's method, rather than the Benjamini-Hochberg FDR adjustment? Latter is strictly more powerful than the former., is adaptive and scales.) | |
| Apr 6, 2021 at 21:27 | comment | added | Gordon Smyth | @Alexis I have not mixed issues. Yes of course power is defined in terms of specified null and alternative. We all understand that. What you have failed to understand is that a set of postdoc hypotheses that span the space of all contrasts combine to match the null and alternative of the F-test. The F-test null is equal to the intersection of the postdoc nulls in the F-test alternative is equal to the union of the posthoc alternatives. It is most certainly possible to conduct a complete analysis of a oneway anova without conducting the F-test. | |
| Apr 6, 2021 at 15:21 | comment | added | Alexis | @GordonSmyth Your rather belabored answer (I downvoted) mixes the issues of Reject ANOVA but Not Reject any post hoc tests and vice versa (a question which has been asked and answered repeatedly on this site, BTW) with some generalized notion of which test is "more powerful". ANOVA and post hoc tests have different null and alternative hypotheses. Since power is literally and explicitly defined with respect to a specific null hypothesis and set of alternative hypotheses, you are comparing apples to oranges. | |
| Apr 6, 2021 at 1:39 | comment | added | Gordon Smyth | @AdamO If there are $G$ groups and $G-1$ linearly independent contrasts are tested then the posthoc tests do test (amongst other things) the overall F-test null hypothesis of no group differences. | |
| Apr 6, 2021 at 1:32 | comment | added | Gordon Smyth | @Alexis The question most certainly is meaningful, indeed it is quite standard. My answer details exactly how the comparison is made and how the posthoc t-tests test (amongst other things) the same null hypothesis as the F-test. | |
| Apr 6, 2021 at 1:08 | history | edited | Alexis | CC BY-SA 4.0 |
edited title
|
| Apr 6, 2021 at 0:34 | comment | added | Alexis | @Sam No, you cannot meaningfully ask which is more powerful, since the definition of powerful is $1 - P(\text{Reject H}_{0} | \text{H}_{0} = \text{False})$. The "$|\text{H}_{0} = \text{False}$" part means the ANOVA power ≠ the post hoc pairwise test power in the same way that apple ≠ orange. | |
| Apr 6, 2021 at 0:31 | history | edited | Alexis | CC BY-SA 4.0 |
deleted 1 character in body; edited title
|
| Apr 5, 2021 at 13:42 | vote | accept | Sam | ||
| Apr 4, 2021 at 9:30 | answer | added | Gordon Smyth | timeline score: 7 | |
| Apr 4, 2021 at 8:31 | comment | added | Sam | I agree, but I think that nevertheless one can still ask which is more powerful. | |
| Apr 2, 2021 at 10:40 | history | edited | kjetil b halvorsen♦ |
edited tags
|
|
| Apr 1, 2021 at 15:32 | comment | added | AdamO | They test different hypotheses. | |
| Apr 1, 2021 at 15:25 | history | asked | Sam | CC BY-SA 4.0 |