0
$\begingroup$

I am repeating a test on a large amount of data and FDR-correcting the p-values afterwards for multiple testing. Yet I still do not have enough power. However, I feel like it is not necessary to test a lot of cases, because I know they will not be significant (let's assume I'm right, because I'm risk-averse). Is it ok to discard these data points before testing in order to improve the overall power, or should the number of data points I dropped still be included when I do the FDR-correction?

Extension: This simple example seems obviously wrong, because we look case by case whether to discard the data or not. Other cases however might be less clear-cut, because they rely on criteria set before, but still use the data.

For example: in the case of observables that are counts, dropping all counts lower than, say 2, because we believe such counts could have happened by chance, and are unrelated to the modeling. Another example: performing the exact same test only once. Say during the multiple testing, we test whether a count of 5 and another one of 4 could have come from the same poisson distribution. Subsequent tests of 5 and 4 have a known answer so we don't include them in the FDR correction.

What I am looking for, is an open-minded yet professional statistician's answer to what is common practice in exploratory analysis of big data, namely play around with the data until we find what is the interesting signal in there, and then design some tests specific to this dataset that give the expected answer (I am of course being very simplistic here). What are accepted practices? What is a no-go? References appreciated.

$\endgroup$
  • $\begingroup$ Professional statisticians recognize the difference between testing hypotheses formulated before the data are viewed (which might not require correction for multiple testing) and post-hoc evaluations of results discovered in the data. They also are careful at keeping exploratory analysis separate from model validation and at testing how well models may apply to the population from which the test sample was taken. Non-professionals who do not proceed with similar caution are typically asking for trouble when they try to use the results of their analyses. $\endgroup$ – EdM Jun 15 '15 at 16:53
  • $\begingroup$ @EdM could you expand on that and make it an answer? $\endgroup$ – yannick Jun 17 '15 at 11:53
1
$\begingroup$

There seem to be two specific issues here. One, from the initial question, is omitting certain comparisons in post-hoc analysis so that there is less of a correction for multiple testing. The other, in the Extension to the original question, is omitting particular observations because of a theoretical consideration.

Neither of these is a good idea. The first has already been addressed by @Cliff AB, so I'll concentrate on the second. Then (not as a professional statistician but as an experienced scientist) I'll comment on the general question about data analysis.

Whenever you drop observations from a sample, you run two risks: your remaining data might no longer represent the broader population that you are trying to understand, and you will lose power for statistical analyses. This is even the case for theory-based decisions like you suggest in the Extension to your original question. I ask: do you really trust your theory more than you trust the data? If you do, why do you need the data at all?

I understand that the drive to make "significant" findings leads to a strong temptation to play with data and models after you have already seen the initial analyses. But if you give in to that temptation, you have a high chance of getting a result that does not apply to the broader population of interest regardless of how well it seems to "fit" the data that you kept.

There are two general ways around this. One is to approach a problem with defined, pre-specified hypotheses and test those hypotheses. Corrections for multiple testing are not generally required for pre-specified hypotheses. Those hypotheses can and should be based on knowledge of the subject matter. If that prior knowledge suggests things like counts that "happened by chance ... unrelated to the modeling," then include those possibilities in the model, don't just throw out data.

The other is to learn from the data in a principled way. Correction for multiple testing is only one of these ways. Other ways use the data that you have in a way that minimizes the chance that your results only apply to your particular sample, not to the population as a whole. Techniques like separating the data into separate training and test sets, cross-validation, and bootstrapping provide this type of principled analysis. Books like An Introduction to Statistical Learning show how these approaches provide reliable ways to test hypotheses generated from the data.

Finally, do not think of this as a one-time-only process. If you care about the validity of your results, you should not stop with analyzing one data set. You should use your initial results to define specific hypotheses and conduct further studies to challenge your hypotheses.

$\endgroup$
5
$\begingroup$

Absolutely not. Dropping the insignificant tests and then pretending that you only considered the tests that ended up being significant is a completely invalid procedure.

If you have, for example, 100 tests and before you see the data you believe that a subset of these tests are going to be more promising than the others, there are various methods you can use to maximize your power.

But looking at your data and only testing the most promising results is cherry-picking, just about to the worst degree.

$\endgroup$
  • $\begingroup$ Thanks! I realize I did not ask the question in the way I intended. I was looking for more general guidelines, see my updated question. $\endgroup$ – yannick Jun 15 '15 at 16:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.