Is the Scheffé test of contrasts the “best-case” for post-hoc tests?

I am reading through Explaining Psychological Statistics, 4th Edition and am stumped after the following passage:

Scheffé (1953) understood that the best anyone can do when creating a complex contrast is to capture all of the $SS_{bet}$ [sum of squared deviations between groups] in a single-df comparison, so that $SS_{bet}$ is divided by 1, instead of $df_{bet}$ [degrees of freedom between groups]. Therefore, in the best-case scenario, $MS_{contrast}$ [mean of squared deviations between groups] equals $df_{bet}$ times $MS_{bet}$, and $F_{contrast}$ [the F ratio for Scheffé's test] equals $df_{bet}$ times $F_{ANOVA}$ [the F ratio in the original ANOVA].

This was right after the section on planned contrasts, and the Scheffé test is described as particularly conservative, so I'm confused. In what sense is Scheffé's test the "best-case"? All I can seem to get out of this and further reading online is that Scheffé's test describes the best-case scenario in a post hoc test but that planned contrasts might have higher power to compensate for their specificity. Is this a correct understanding of affairs?

Firstly, there are many tests and each one has its good and bad sides over another test. Some of the most basic tests are Tukey's, Bonferroni's and Scheffés, so we can compare and contrast these to understand what goes into choosing a test.

A useful explanation of choosing between tests (http://www.itl.nist.gov/div898/handbook/prc/section4/prc473.htm):

1. If all pairwise comparisons are of interest, Tukey has the edge. If only a subset of pairwise comparisons are required, Bonferroni may sometimes be better.

2. When the number of contrasts to be estimated is small, (about as many as there are factors) Bonferroni is better than Scheffé. Actually, unless the number of desired contrasts is at least twice the number of factors, Scheffé will always show wider confidence bands than Bonferroni.

3. Many computer packages include all three methods. So, study the output and select the method with the smallest confidence band. edit: (This is listed in the source but is likely not a good idea).

4. No single method of multiple comparisons is uniformly best among all the methods.

Here is a simplified decision tree for choosing the correct test (http://www.statsdirect.com/help/content/analysis_of_variance/multiple_comparisons.htm):

• pairwise

• equal group sizes: Tukey
• unequal group sizes: Tukey-Kramer or Scheffé's
• not pairwise

• planned: Bonferroni
• not planned: Scheffé's

Tangent:

In addition, you need to understand the difference between planned and unplanned comparisons. If you know every comparison that you want to compare then you can use a planned comparison test like Tukey and Bonferroni. If you think you may need to do some of the notorious data snooping then you can at least adjust for the unplanned comparisons with Scheffé's method. (Tukey: http://www.itl.nist.gov/div898/handbook/prc/section4/prc471.htm, Bonferroni: http://www.itl.nist.gov/div898/handbook/prc/section4/prc473.htm, Scheffé: http://www.itl.nist.gov/div898/handbook/prc/section4/prc472.htm)

In general, do not data snoop (unplanned comparisons) if possible. You need to understand what you are looking for (planned comparisons) and should consult with a professional statistician before deciding what test would be best. Also, an understanding of the data you are testing will allow you to do specific comparisons and thus allow you to use a specific test that gives you the most narrow confidence interval (detect differences) while being statistically correct.

• Well I'm afraid I don't have a professional statistician and can't afford one and I'm doing this all on my own :) – readyready15728 Jun 29 '16 at 17:30
• But just so we're clear: Scheffé is strictly a post hoc test, and that's the only situation it can possibly be the best in? I will give you the bounty once that's cleared up. – readyready15728 Jun 29 '16 at 17:33
• Scheffe is strictly a post hoc test, so that's the only situation it can possibly be used in. If you desired a test that is not post hoc then you would use t-test, F-test, contrasts, etc. It may help to read more about the difference between planned and unplanned comparisons. Mainly, why post hoc tests are used. – Travis Jun 29 '16 at 20:51
• @Travis I like your answer but I don't agree that Scheffe is strictly a post hoc test. The idea of his approach is to find a confidence interval for all possible values of contrast "including those chosen after looking at the data" (Rencher & Schallje, "Linear Models in Statistics",2nd ed., page 207). This happens because the intervals are constructed based on the maximum value taken by the test statistic (that's why it's conservative). – Gus_est Jul 1 '16 at 2:55
• @Travis Also, as you said: "When the number of contrasts to be estimated is small, (about as many as there are factors) Bonferroni is better than Scheffé". This is true, but on the other hand, for a large number of constrasts, the Scheffe test is better since its critical value doesn't change with the number of comparisons (as opposed to the critical value for Bonferroni tests, which increases with the number of tests and "eventually exceeds the critical value for Scheffe tests" (same book, page 209). – Gus_est Jul 1 '16 at 2:59