Am I Double-Dipping on my Hypothesis Test? I am testing a large number of simultaneous hypotheses (~300), with each hypothesis test trying to determine whether a given test statistic differs significantly from zero.
Am I cheating the multiple hypothesis framework if I first throw out all test statistics whose estimates are obviously very close to zero, and then run my multiple hypothesis test (with alpha=0.05) on the remainder?
For example, suppose I started with 300 test statistics with values ranging from -8 to 8. Suppose further that 200 of these statistics have values in the range [-1,1]. Would it then be acceptable to throw away these 200 test statistics (which I know will not reach significance in any subsequent hypothesis test), and only test the remaining 100 statistics with a Bonferroni-corrected p-value cutoff of $\frac{0.05}{100}$ (instead of $\frac{0.05}{300}$)? Or am I double-dipping here?
 A: With each statistic, you're still checking whether it would exceed the critical value. For a given test statistic, knowing that it doesn't exceed the critical value because its absolute value is less than 1 doesn't alter the fact that you looked at its size and figured out whether that statistic fell in the rejection region. 
Which is to say, you still did a test.
Consider the following scenario. We have a large collection of t-statistics (say 100 of them), all with df between 55 and 60, on each of which we need to perform a two-tailed t-test. We each take the collection of statistics and find out which ones were rejections:
A) For each test statistic, I find the relevant critical value and either reject or fail to reject the null. Let's say I decided "not reject" on 30 of them and "reject" on 70 of them. This is how things normally work, and clearly I did 100 tests.
B) You happen know that the critical value for $\nu=60$ is above 2, and it's higher still for $\nu<60$. So first you pull out all the cases with $|t|<2$ and put those aside; you don't even calculate the corresponding critical value for those cases because clearly they can't exceed it. Now for the remaining 71 cases you calculate the critical value and reject 70 of them. 
When you took approach B did you really do 29 fewer tests than I did under scheme A? You still started with 100 test statistics, and you still classified them as 'reject' or 'not reject' -- you just split that decision up into two stages -- (i) don't reject any with $|t|<2$, and (ii) check to see which of the remaining cases are rejections.
[Or consider it this way: we know the first approach is correct. If you come up with any scheme that organizes the same work differently and it comes up with a different count of how many tests you did, you must have counted wrong.]
