0
$\begingroup$

I have a list of say 50,000 e-values (multiple testing corrected P values) for a statistical test I am doing that essentially matches patterns together. Essentially the e-value is the expectation that these two patterns would match purely by chance.

Most people who do this will do another round of multiple testing correction on their list of e-values which are already corrected for the number of tests that generate that e-value. For example each E-value may be the result of 100,000 tests matching different patterns together. The theory is when you have 50,000 of these e-values you will still have a good number of type I error's associated with your list.

Can someone give me a some theory on why it's a good idea to apply multiple testing correction to values that were already corrected for multiple testing?

Is there a good paper I can read that will help me understand? Conceptually I can understand why you want to do this, because that list of 50,000 e-vales will have some false positives, but if there is a paper someone can point me to that would be great.

$\endgroup$
1
$\begingroup$

A possible solution to your inquiry is to use the Simes method for multiple testing correction

(Simes, R. J. (1986): An improved Bonferroni procedure for multiple tests of significance. Biometrika 73 (3): 751-754)

which can be found at http://www-stat.wharton.upenn.edu/~steele/Courses/956/Resource/MultipleComparision/Simes86pdf.pdf

Briefly, Simes suggesting using an alternative Bonferroni-like procedure, where the corrected p-value is given by

$p_{s}=\underset{k}{min}\: mp_{(k)} /k$

where $m$ is the number of tests performed (50000 in your example) and $k$ is the number of times the test was applied (100000 in your case if I am not mistaken?). Keep in mind that there might be p-value dependence issues but Simes argues that from numerical evidence this correction still performs well. Have a look also at https://www.researchgate.net/publication/2255729_Combining_Statistical_Tests_By_Multiplying_p-values. This the source of many parts of the anwser and I have applied the Simes solution myself regarding the application of different statistical tests on the same data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.