# How does Scheffe test deal with data snooping?

hypotheses suggested by the data, if tested using the data set that suggested them, are likely to be accepted even when they are not true. This is because circular reasoning would be involved: something seems true in the limited data set, therefore we hypothesize that it is true in general, therefore we (wrongly) test it on the same limited data set, which seems to confirm that it is true. Generating hypotheses based on data already observed, in the absence of testing them on new data, is referred to as post hoc theorizing.

The correct procedure is to test any hypothesis on a data set that was not used to generate the hypothesis.

For Post Hoc analysis of ANOVA,

Henry Scheffé's simultaneous test of all contrasts in multiple comparison problems is the most[citation needed] well-known remedy in the case of analysis of variance.1 It is a method designed for testing hypotheses suggested by the data while avoiding the fallacy described above.

So I was wondering how Scheffé's test avoids the fallacy of data snooping?

• The Wikipedia article you link to states "It can be shown that the probability is $1-\alpha$ that all confidence limits of the type $$\hat{C}\pm s_\hat{C}\sqrt{(r-1)F_{\alpha;r-1;N-r}}$$ are simultaneously correct." That seems like a particularly clear and full answer to me. Do you detect that something is missing in it?
– whuber
Jul 11, 2013 at 1:45
• @whuber: Thanks! (1) Why does the quote in your comment explains Scheffe's test avoids the data snooping? (2) Also does Bonferroni multiple comparison procedure not avoid data snooping problem, and why? See my question here.
– Tim
Jul 11, 2013 at 1:52
• It seems to me that the quotation with which you begin your previous question nicely answers both (1) and (2). The distinction it explicitly draws is between a "family of inferences" specified in advance and a family that is not, or cannot, be so specified. The Wikipedia quotation is in the context of a clearly defined family: namely, all possible contrasts.
– whuber
Jul 11, 2013 at 1:55
• I think this is a beautiful and important question! There is a paper from Berk et al. (2013) that could be interesting for you (the title is 'Valid Post-Selection Inference'). They develop confidence intervals that are robust to any kind of selection/snooping/significance searching, given normal errors and a fixed design. They manage to derive PoSI (Post selection inference) constants that serve as critical values. They are (in absolute value) upper bounded by the Scheffé constant, demonstrating that it in fact provides a 'guarantee against data snooping'. Feb 16, 2016 at 15:46

The Wikipedia article you link to states "It can be shown that the probability is $1−α$ that all confidence limits of the type $$\hat{C}\pm s_\hat{C}\sqrt{(r-1)F_{\alpha;r-1;N-r}}$$ are simultaneously correct." That seems like a particularly clear and full answer to me. Do you detect that something is missing in it? – whuber