# Purpose of Scheffé's confidence intervals

Scheffé's method, when first written about by Henry Scheffé, was described as serving the purpose of identifying statistically significant contrasts in multiple-comparison's ANOVA problems.

But here in this forum I am told in answers and comments under this question that that should not even be done:

"A test of contrasts will only have the correct distribution under the null when the contrasts are specified before you see the data."

As stated, that is of course nonsense. It depends on what test is used. If you use at test designed only for testing whether a particular contrast differs significantly from zero, and you find that there are some contrasts that are significant, then you're making a mistake. But that's what I take Scheffé's method to be intended to remedy.

And in a comment:

"And in general it's unwise to even consider testing patterns in data based solely on the pattern in the data."

What is Scheffé's method for if not exactly that? It seems to me that some people have been instructed that testing a contrast suggested by the data is not valid if one uses the same test one would use if that particular contrast had been identified in advance (which is obviously right) and leapt from there to the conclusion that there are no correct methods of testing contrasts suggested by the data.

"The naive method is incorrect; therefore no method is correct." ---- my paraphrase

Could those who want to defend the view I'm paraphrasing explain what they think the purpose of Scheffé's method is?

This is a classic case of simultaneous inference used for selective inference (see [1]). I will explain.

"Selective (marginal) inference" is when you want to make inference on a data-driven subset of parameters. In the ANOVA case, the subset would be several contrasts. "Simultaneous inference" is when you want to make inference on a particular vector of parameters. Naturally, simultaneous entails selective, as the joint truthfulness of a vector, entails the truthfulness of each of its (selected) elements.

Sheffe's view of the problem is the following: since he does not know a-priori the contrast the researcher will study, he will offer simultaneous control over all possible contrasts. In this case, no matter what contrast the researcher chooses, data driven or not, he is already controlling for it.

In conclusion: if using Scheffe's method for inference or CI, there is no problem in inferring on selected parameters. The problem is that for a particular parameter/contrast, it is excessively conservative (i.e., low powered).

[1] Cox, D. R. 1965. “A Remark on Multiple Comparison Methods.” Technometrics 7 (2): 223–224. doi:10.1080/00401706.1965.10490250.

"A test of contrasts will only have the correct distribution under the null when the set of contrasts that might be tested is specified before you see the data." would have been a better way of putting it. Scheffé's method then specifies all possible contrasts between the means of pre-defined groups & I doubt there's any remaining disagreement.

I think the point here was that you seemed to be proposing the application of a new contrast-based test suggested by particular data to that data itself; and the experimentwise error rate is not really controlled in such a situation because the set of tests you might have invented is not well defined, even though once you've defined the test the set of contrasts is.