Dear community members,

I performed thousands of tests, each of which do some regression comparison between numbers of events in cases and controls. Each test comprised thousands of cases, a 3-fold imbalance favoring controls, and events typically among 10-90 samples per test. The hypothesis in each test is whether events in cases are more frequent than in controls. After I perform all these tests, I have to correct for the number of tests I performed to control the FDR for all the comparisons.

However, it seems that I have to performed too many tests. Specifically, I want to remove tests for which they would never be significant under any circumstances. For example, if only couple of events happened in both cases and controls, I can remove this test without performing actual comparison.

In general, if I understand correctly, I can remove tests with any criteria which does not take differences between cases and controls into account and the distribution of $p$-values should be fine for multiplicity correction methods, in other words the distribution of the $p$-values will be approximately uniform.

  1. Is it true?
  2. Can I remove tests that have less than X events in cases (not looking at controls at all)? Will I artificially "deflate" the right end of the p-value distribution by doing so and should I care about the right end of the p-values distribution (I use standard FDR correction procedures)?

Update: I was finally able to Google the idea I want to fully understand - it is called Tarone's exclusion principle ( https://www.jstor.org/stable/3109773?seq=1#metadata_info_tab_contents ). Since this work was done quite a long time ago, I am looking into the updates on this topic. Are the findings and perspectives here mostly unchanged since the initial publication?


1 Answer 1


You did not know a priori that the specific analysis would render insufficient numbers of cases, or else you would have excluded the tests before you ever collected the data. Alternately, consider if you did the experiment again, you can't be certain that that specific test would meet the criteria for exclusion again. In some large dimensional analyses I have seen the FDR set to a somewhat less conservative value to ensure that promising signals aren't excluded.

Just a comment about blindedness: the fact that you have already observed that some of the cases which can "never reach significance" include "only couple of events happened in both cases and controls" means you are not qualified to apply any blinded exclusion even if automated.

Further, regarding conservation of the nominal 0.05 alpha rate: FDR methods do not correct the $p$-values for specific analyses, they simply reclassify a handful of significant analyses as "non-significant" based on the expected number of false positives given the number of tests performed. Tarone's method handles Bonferroni procedures. I think you are mixing up control of FWER versus FDR. You can use FWER methods like Bonferroni (with Tarone's method) to control FDR but it will be WAY more conservative.

  • $\begingroup$ Absolutely true, but when I collected the data, such tests won't have even a simple p-value < 0.05 (there is no possibility, e.g. lady tasting tea problem but with 3 vs 3 cups) - I see them as simply inflating the distribution with non-informative results... It seems for me that whatever they are (H0 or HA) - removal of them will be only beneficial for the "proportion of positive tests" FDR estimations, or can it spoil FDR estimations? $\endgroup$ Feb 26, 2021 at 15:56
  • $\begingroup$ I think it was called Tarone's exclusion principle - jstor.org/stable/3109773?seq=1#metadata_info_tab_contents - I want to understand up to which degree it can be expanded... $\endgroup$ Feb 26, 2021 at 16:08
  • 1
    $\begingroup$ @GermanDemidov see edit. $\endgroup$
    – AdamO
    Feb 26, 2021 at 16:39
  • $\begingroup$ thanks a lot, it became much clearer! I have problems with expressing statistical thoughts... $\endgroup$ Feb 26, 2021 at 17:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.