Increasing the power by dropping points: can I do it?

Question

I am repeating a test on a large amount of data and FDR-correcting the p-values afterwards for multiple testing. Yet I still do not have enough power. However, I feel like it is not necessary to test a lot of cases, because I know they will not be significant (let's assume I'm right, because I'm risk-averse). Is it ok to discard these data points before testing in order to improve the overall power, or should the number of data points I dropped still be included when I do the FDR-correction?

Extension: This simple example seems obviously wrong, because we look case by case whether to discard the data or not. Other cases however might be less clear-cut, because they rely on criteria set before, but still use the data.

For example: in the case of observables that are counts, dropping all counts lower than, say 2, because we believe such counts could have happened by chance, and are unrelated to the modeling. Another example: performing the exact same test only once. Say during the multiple testing, we test whether a count of 5 and another one of 4 could have come from the same poisson distribution. Subsequent tests of 5 and 4 have a known answer so we don't include them in the FDR correction.

What I am looking for, is an open-minded yet professional statistician's answer to what is common practice in exploratory analysis of big data, namely play around with the data until we find what is the interesting signal in there, and then design some tests specific to this dataset that give the expected answer (I am of course being very simplistic here). What are accepted practices? What is a no-go? References appreciated.

Answer 1

There seem to be two specific issues here. One, from the initial question, is omitting certain comparisons in post-hoc analysis so that there is less of a correction for multiple testing. The other, in the Extension to the original question, is omitting particular observations because of a theoretical consideration.

Neither of these is a good idea. The first has already been addressed by @Cliff AB, so I'll concentrate on the second. Then (not as a professional statistician but as an experienced scientist) I'll comment on the general question about data analysis.

Whenever you drop observations from a sample, you run two risks: your remaining data might no longer represent the broader population that you are trying to understand, and you will lose power for statistical analyses. This is even the case for theory-based decisions like you suggest in the Extension to your original question. I ask: do you really trust your theory more than you trust the data? If you do, why do you need the data at all?

I understand that the drive to make "significant" findings leads to a strong temptation to play with data and models after you have already seen the initial analyses. But if you give in to that temptation, you have a high chance of getting a result that does not apply to the broader population of interest regardless of how well it seems to "fit" the data that you kept.

There are two general ways around this. One is to approach a problem with defined, pre-specified hypotheses and test those hypotheses. That reduces the number of corrections for multiple testing. Those hypotheses can and should be based on knowledge of the subject matter. If that prior knowledge suggests things like counts that "happened by chance ... unrelated to the modeling," then include those possibilities in the model, don't just throw out data.

The other is to learn from the data in a principled way. Correction for multiple testing is only one of these ways. Other ways use the data that you have in a way that minimizes the chance that your results only apply to your particular sample, not to the population as a whole. Techniques like separating the data into separate training and test sets, cross-validation, and bootstrapping provide this type of principled analysis. Books like An Introduction to Statistical Learning show how these approaches provide reliable ways to test hypotheses generated from the data.

Finally, do not think of this as a one-time-only process. If you care about the validity of your results, you should not stop with analyzing one data set. You should use your initial results to define specific hypotheses and conduct further studies to challenge your hypotheses.

Answer 2

Absolutely not. Dropping the insignificant tests and then pretending that you only considered the tests that ended up being significant is a completely invalid procedure.

If you have, for example, 100 tests and before you see the data you believe that a subset of these tests are going to be more promising than the others, there are various methods you can use to maximize your power.

But looking at your data and only testing the most promising results is cherry-picking, just about to the worst degree.

Answer 3

Your question might relate to 'independent filtering' of data that is performed by commonly used bioinformatics pipelines before any FDR adjustment takes place.

The typical situation this is applied is when there are a very large number of outcomes to be tested (say expression levels of different genes) and a small number of predictors.

The aim of this, as you suggest, is to reduce the total number of hypotheses to be tested. The software will remove outcomes where there is insufficient data on that outcome for a positive signal to be detected. That is, we can tell that the power for that outcome is going to be very very low, irrespective of the true effect size, so this outcome can be ignored. It does not use any information from the predictors in this process.

A critereon for gene expression might be that for a gene to be carried forward into FDR adjustment and interpretation phase, a certain number of samples in the dataset must have non-zero expression levels, or that the average expression level (or variation in expression level) should exceed a certain threshold.

If thresholds are decided a priori I don't see a problem with this approach. The approach outlined in the link below goes a bit further, tuning the threshold to obtain the maximum number of 'significant' results following the FDR adjustment from the remaining dataset. This feels like it could be problematic, I haven't explored the arguments for why it might or might not be.

See here for more information with respect to gene expression analysis with Deseq2. https://uclouvain-cbio.github.io/WSBIM2122/sec-rnaseq.html#independent-filtering