0
$\begingroup$

Assuming I have 100 variables and I do a t-test for each of them without adjusting the p-values, and get 8 significant p-values out of the 100 tests. If my alpha is 5% can I just say that since I expect by chance 5% of p-values to be under 0.05, the 5 highest p-values for those 8 significant tests are due to random chance (since 5 is the expected 5% of 100. And then say that the 3 lowest p-values are the true rejections of the null hypothesis and are adjusted for multiple testing (and therefore my analysis adjusts for multiple tetsing)?

$\endgroup$
3
$\begingroup$

No, you do not control the familywise type one error rate this way, with sufficiently many tests you end up rejecting at least one null hypothesis about half of the time under the global null hypothesis (50% type 1 error rate rather than 5%). This procedure also has the odd property that you can never reject all null hypotheses no matter how strong your evidence is.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.