0
$\begingroup$

From Kutner's Applied Linear Statistical Models

The Bonferroni multiple comparison procedure does not lend itself to data snooping, unless one can specify in advance the family of inferences in which one may be interested and provided this family is not large.

The Tukey and Scheffe procedures involve families of inferences that lend themselves naturally to data snooping. the Tukey and Scheffe procedures, allow data snooping to be undertaken naturally without affecting the confidence coefficient or significance level.

Why is Bonferroni procedure not working well in data snooping problem, while the Tukey and Scheffe procedures can?

How shall I understand the reason of "unless one can specify in advance the family of inferences in which one may be interested and provided this family is not large"?

Thanks!

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ possible duplicate of Why is the Bonferroni procedure applicable only when the effects to be investigated are identified in advance of the study? $\endgroup$
    – Scortchi ♦
    Commented Jul 8, 2013 at 21:44
  • 1
    $\begingroup$ I think if you shift the focus of your question to the second part "... and provided this family is not large" you'd have a distinct question. $\endgroup$
    – russellpierce
    Commented Jul 8, 2013 at 23:27
  • $\begingroup$ @Scortchi: The two posts are different questions. Here I am asking about data snooping, and there I asked about why effects must be identified for Bonferroni procedures. They are from the same book, but not the same questions. $\endgroup$
    – Tim
    Commented Jul 8, 2013 at 23:32

1 Answer 1

Reset to default
3
$\begingroup$

Two reasons they might have said "[...] and provided this family is not large" are that when it is:–

(1) The Bonferroni procedure gives a conservative bound, & you could perhaps get higher power using the Scheffé or Tukey's HSD procedures even if you don't want to test every contrast or pairwise difference.

(2) Controlling the false discovery rate may make more sense.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thansk, but data snooping means something like overfitting problem $\endgroup$
    – Tim
    Commented Jul 9, 2013 at 11:09
  • $\begingroup$ Read the link you just posted: "When large numbers of tests are performed, some produce false results, hence 5% of randomly chosen hypotheses turn out to be significant at the 5% level, 1% turn out to be significant at the 1% significance level, and so on, by chance alone. This and a comic example exemplify the multiple comparisons hazard in data dredging." $\endgroup$
    – Scortchi ♦
    Commented Jul 9, 2013 at 11:22
  • $\begingroup$ In any case if you think this answer doesn't apply for some reason, please explain why not. IMO the only substantive difference between the two questions is the one @Russell pointed out. $\endgroup$
    – Scortchi ♦
    Commented Jul 9, 2013 at 12:01
  • $\begingroup$ @Tim Usually it does but clearly, given the tone of the second paragraph, the authors don't mean anything bad by it in this case, probably more something like “exploration“, which explains why Bonferroni would seem less appropriate because it would entail a steep power decrease. $\endgroup$
    – Gala
    Commented Jul 9, 2013 at 12:08
  • 1
    $\begingroup$ In general, I think relaxing a bit your view of language would help understanding (e.g. accepting that there isn't necessarily one true universal definition for each word, realizing that many things are written quickly without pondering every nuance of every sentence for hours and that people occasionally use words, even technical ones, in slightly different ways). In short, don't read too much into every text you come across. $\endgroup$
    – Gala
    Commented Jul 9, 2013 at 12:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.