# What do “single-step” and “multi-step” mean in post-hoc testing of ANOVAs?

From Wikipedia

Post-hoc testing of ANOVAs

Multiple comparison procedures are commonly used in an analysis of variance after obtaining a significant omnibus test result, like the ANOVA F-test. The significant ANOVA result suggests rejecting the global null hypothesis H0 that the means are the same across the groups being compared. Multiple comparison procedures are then used to determine which means differ. In a one-way ANOVA involving K group means, there are K(K − 1)/2 pairwise comparisons.

A number of methods have been proposed for this problem, some of which are:

Single-step procedures

• Tukey–Kramer method (Tukey's HSD) (1951)
• Scheffe method (1953)

Multi-step procedures based on Studentized range statistic

• Duncan's new multiple range test (1955)
• The Nemenyi test is similar to Tukey's range test in ANOVA.

• The Bonferroni–Dunn test allows comparisons, controlling the familywise error rate.[vague]

• Student Newman-Keuls post-hoc analysis
• Dunnett's test (1955) for comparison of number of treatments to a single control group.

I was wondering what "single-step" and "multi-step" mean here?

Are they both for pairwise comparisons for every two groups, right?

Thanks and regards!

Single-step and step-wise relate to the view of the procedures as dynamic.

A single-step procedure implies there is no dynamics: without looking at the data, the procedure offers some rejection threshold.

A step-wise procedure implies there is dynamics: rejection boundaries are data driven and are updated along the sequence of p-values/test statistics in the data.

In reality, there is no real dynamics, as even the step-wise procedures take all test statistics, and returns a rejection boundary. The name stems mainly from the motivation to the procedure.

• Thanks! (1) "updated along the sequence of p-values/test statistics in the data", does each in the "sequence" means for each individual test in the multiple comparison? (2) A side question: are the above single-step and multi-step procedures all parametric, e.g. assuming the samples are normally distributed? – Tim May 26 '13 at 12:58
• (1) each member of a "sequence" is a test statistic. (2) you need to assume the marginal distribution of each test statistic. – JohnRos May 27 '13 at 5:37
• Thanks, JohnRos! Since "In reality, there is no real dynamics, as even the step-wise procedures take all test statistics, and returns a rejection boundary", the rejection boundary must be specified completely before seeing the data, so how shall I understand "rejection boundaries are data driven and are updated along the sequence of p-values/test statistics in the data"? – Tim Jul 8 '13 at 19:05
• I think I disagree with "the rejection boundary must be specified completely before seeing the data". – JohnRos Jul 8 '13 at 20:06
• as a testing rule, isn't its rejection region completely specified before seeing any data? Do you mean the rejection region can change for different data? That will not be called testing. – Tim Jul 8 '13 at 20:12